Systems and Methods for Estimating Depth and Visibility from a Reference Viewpoint for Pixels in a Set of Images Captured from Different Viewpoints

ABSTRACT

Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates.

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application claims priority as a continuation of U.S. patentapplication Ser. No. 14/526,392 filed Oct. 28, 2014, which is acontinuation of U.S. patent application Ser. No. 14/329,754 entitled“Systems and Methods for Estimating Depth and Visibility from aReference Viewpoint for Pixels in a Set of Images Captured fromDifferent Viewpoints”, filed Jul. 11, 2014, which is a continuation ofU.S. patent application Ser. No. 14/144,458 entitled “Systems andMethods for Performing Depth Estimation using Image Data from MultipleSpectral Channels”, filed Dec. 30, 2013, which is a continuation of U.S.patent application Ser. No. 13/972,881 entitled “Systems and Methods forParallax Detection and Correction in Images Captured Using Array Camerasthat Contain Occlusions using Subsets of Images to Perform DepthEstimation”, filed Aug. 21, 2013, which claims priority to U.S.Provisional Patent Application Ser. No. 61/691,666 to Venkataraman etal. entitled “Systems and Methods for Parallax Detection and Correctionin Images Captured using Array Cameras”, filed Aug. 21, 2012 and U.S.Provisional Patent Application Ser. No. 61/780,906 to Venkataraman etal. entitled “Systems and Methods for Parallax Detection and Correctionin Images Captured using Array Cameras”, filed Mar. 13, 2013. Thedisclosures of U.S. patent application Ser. Nos. 14/329,754, 14/144,458,13/972,881 and U.S. Provisional Patent Application Ser. No. 61/691,666and 61/780,906 are hereby incorporated by reference herein in theirentirety.

FIELD OF THE INVENTION

The present invention generally relates to digital cameras and morespecifically to the detection and correction of parallax in imagescaptured using array cameras.

BACKGROUND

Binocular viewing of a scene creates two slightly different images ofthe scene due to the different fields of view of each eye. Thesedifferences, referred to as binocular disparity (or parallax), provideinformation that can be used to calculate depth in the visual scene,providing a major means of depth perception. The impression of depthassociated with stereoscopic depth perception can also be obtained underother conditions, such as when an observer views a scene with only oneeye while moving. The observed parallax can be utilized to obtain depthinformation for objects in the scene. Similar principles in machinevision can be used to gather depth information.

Two or more cameras separated by a distance can take pictures of thesame scene and the captured images can be compared by shifting thepixels of two or more images to find parts of the images that match. Theamount an object shifts between different camera views is called thedisparity, which is inversely proportional to the distance to theobject. A disparity search that detects the shift of an object inmultiple images can be used to calculate the distance to the objectbased upon the baseline distance between the cameras and the focallength of the cameras involved. The approach of using two or morecameras to generate stereoscopic three-dimensional images is commonlyreferred to as multi-view stereo.

Multi-view stereo can generally be described in terms of the followingcomponents: matching criterion, aggregation method, and winnerselection. The matching criterion is used as a means of measuring thesimilarity of pixels or regions across different images. A typical errormeasure is the RGB or intensity difference between images (thesedifferences can be squared, or robust measures can be used). Somemethods compute subpixel disparities by computing the analytic minimumof the local error surface or use gradient-based techniques. One methodinvolves taking the minimum difference between a pixel in one image andthe interpolated intensity function in the other image. The aggregationmethod refers to the manner in which the error function over the searchspace is computed or accumulated. The most direct way is to apply searchwindows of a fixed size over a prescribed disparity space for multiplecameras. Others use adaptive windows, shiftable windows, or multiplemasks. Another set of methods accumulates votes in 3D space, e.g., aspace sweep approach and voxel coloring and its variants. Once theinitial or aggregated matching costs have been computed, a decision ismade as to the correct disparity assignment for each pixel. Localmethods do this at each pixel independently, typically by picking thedisparity with the minimum aggregated value. Cooperative/competitivealgorithms can be used to iteratively decide on the best assignments.Dynamic programming can be used for computing depths associated withedge features or general intensity similarity matches. These approachescan take advantage of one-dimensional ordering constraints along theepipolar line to handle depth discontinuities and unmatched regions. Yetanother class of methods formulate stereo matching as a globaloptimization problem, which can be solved by global methods such assimulated annealing and graph cuts.

More recently, researches have used multiple cameras spanning a widersynthetic aperture to capture light field images (e.g. the StanfordMulti-Camera Array). A light field, which is often defined as a 4Dfunction characterizing the light from all direction at all points in ascene, can be interpreted as a two-dimensional (2D) collection of 2Dimages of a scene. Due to practical constraints, it is typicallydifficult to simultaneously capture the collection of 2D images of ascene that form a light field. However, the closer in time at which theimage data is captured by each of the cameras, the less likely thatvariations in light intensity (e.g. the otherwise imperceptible flickerof fluorescent lights) or object motion will result in time dependentvariations between the captured images. Processes involving capturingand resampling a light field can be utilized to simulate cameras withlarge apertures. For example, an array of M×N cameras pointing at ascene can simulate the focusing effects of a lens as large as the array.Use of camera arrays in this way can be referred to as syntheticaperture photography.

While stereo matching was originally formulated as the recovery of 3Dshape from a pair of images, a light field captured using a camera arraycan also be used to reconstruct a 3D shape using similar algorithms tothose used in stereo matching. The challenge, as more images are added,is that the prevalence of partially occluded regions (pixels visible insome but not all images) also increases.

SUMMARY OF THE INVENTION

Systems and methods in accordance with embodiments of the invention canperform parallax detection and correction in images captured using arraycameras. An embodiment of the method of the invention for estimatingdistances to objects within a scene from a light field comprising a setof images captured from different viewpoints using a processorconfigured by an image processing application includes: selecting areference viewpoint relative to the viewpoints of the set of imagescaptured from different viewpoints; normalizing the set of images toincrease the similarity of corresponding pixels within the set ofimages; and determining initial depth estimates for pixel locations inan image from the reference viewpoint using at least a subset of the setof images, where an initial depth estimate for a given pixel location inthe image from the reference viewpoint is determined by: identifyingpixels in the at least a subset of the set of images that correspond tothe given pixel location in the image from the reference viewpoint basedupon expected disparity at a plurality of depths; comparing thesimilarity of the corresponding pixels identified at each of theplurality of depths; and selecting the depth from the plurality ofdepths at which the identified corresponding pixels have the highestdegree of similarity as an initial depth estimate for the given pixellocation in the image from the reference viewpoint. In addition, themethod includes identifying corresponding pixels in the set of imagesusing the initial depth estimates; comparing the similarity of thecorresponding pixels in the set of images to detect mismatched pixels.When an initial depth estimate does not result in the detection of amismatch between corresponding pixels in the set of images, selectingthe initial depth estimate as the current depth estimate for the pixellocation in the image from the reference viewpoint. When an initialdepth estimate results in the detection of a mismatch betweencorresponding pixels in the set of images, selecting the current depthestimate for the pixel location in the image from the referenceviewpoint by: determining a set of candidate depth estimates using aplurality of different subsets of the set of images; identifyingcorresponding pixels in each of the plurality of subsets of the set ofimages based upon the candidate depth estimates; and selecting thecandidate depth of the subset having the most similar correspondingpixels as the current depth estimate for the pixel location in the imagefrom the reference viewpoint.

In a further embodiment, selecting a reference viewpoint relative to theviewpoints of the set of images captured from different viewpointsincludes selecting a viewpoint from the set consisting of: the viewpointof one of the images; and a virtual viewpoint.

In another embodiment, a pixel in a given image from the set of imagesthat corresponds to a pixel location in the image from the referenceviewpoint is determined by applying a scene dependent shift to the pixellocation in the image from the reference viewpoint that is determinedbased upon: the depth estimate of the pixel location in the image fromthe reference viewpoint; and the baseline between the viewpoint of thegiven image and the reference viewpoint.

In a still further embodiment, the subsets of the set of images used todetermine the set of candidate depth estimates are selected based uponthe viewpoints of the images in the sets of images to exploit patternsof visibility characteristic of natural scenes that are likely to resultin at least one subset in which a given pixel location in the image fromthe reference viewpoint is visible in each image in the subset.

In still another embodiment, the set of images are captured withinmultiple color channels; selecting a reference viewpoint relative to theviewpoints of the set of images captured from different viewpointsincludes selecting one of the images as a reference image and selectingthe viewpoint of the reference image as the reference viewpoint; and thesubsets of the set of images used to determine the set of candidatedepth estimates are selected so that the same number of images in thecolor channel containing the reference image appears in each subset.

In a yet further embodiment, the subsets of the set of images used todetermine the set of candidate depth estimates are also selected so thatthere are at least two images in the color channels that do not containthe reference image in each subset.

Yet another embodiment also includes determining the visibility of thepixels in the set of images from the reference viewpoint by: identifyingcorresponding pixels in the set of images using the current depthestimates; and determining that a pixel in a given image is not visiblein the image from the reference viewpoint when the pixel fails aphotometric similarity criterion determined based upon a comparison ofcorresponding pixels.

In a further embodiment again, selecting a reference viewpoint relativeto the viewpoints of the set of images captured from differentviewpoints includes selecting one of the images in the set of images asa reference image and selecting the viewpoint of the reference image asthe reference viewpoint; and determining that a pixel in a given imageis not visible in the image from the reference viewpoint when the pixelfails a photometric similarity criterion determined based upon acomparison of corresponding pixels further includes comparing the pixelin the given image to the corresponding pixel in the reference image.

In another embodiment again, the photometric similarity criterionincludes a similarity threshold that adapts based upon at least theintensity of at least one of the pixel in the given image and the pixelin the reference image.

In a further additional embodiment, the photometric similarity criterionincludes a similarity threshold that adapts as a function of thephotometric distance between the corresponding pixel from the referenceimage and the corresponding pixel that is most similar to the pixel fromthe reference image.

In another additional embodiment, the photometric similarity criterionincludes a similarity threshold that adapts based upon the signal tonoise ratio of the pixel in the reference image.

In a still yet further embodiment, adapting the similarity thresholdbased upon the signal to noise ratio is approximated by scaling thephotometric distance of the corresponding pixel from the reference imageand the corresponding pixel that is most similar to the pixel from thereference image is and applying an offset to obtain an appropriatethreshold.

In still yet another embodiment, the set of images includes imagescaptured in a plurality of color channels and the reference image is animage captured in a first color channel and the given image is in thesecond color channel; determining that a pixel in a given image is notvisible in the reference viewpoint when the pixel fails a photometricsimilarity criterion determined based upon a comparison of correspondingpixels further includes: selecting an image in the second color channelin which the corresponding pixel in the image from the referenceviewpoint is visible as a reference image for the second color channel;and comparing the pixel in the given image to the corresponding pixel inthe reference image for the second color channel.

In a still further embodiment again, selecting a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints includes selecting a virtual viewpoint as the referenceviewpoint; and determining that a pixel in a given image is not visiblein the image from the reference viewpoint when the pixel fails aphotometric similarity criterion determined based upon a comparison ofcorresponding pixels further includes: selecting an image adjacent thevirtual viewpoint as a reference image; and comparing the pixel in thegiven image to the corresponding pixel in the reference image.

In still another embodiment again, the image adjacent the virtualviewpoint is selected based upon the corresponding pixel in the selectedimage to the pixel from the given image being visible in an image fromthe reference viewpoint.

A yet further embodiment again also includes updating the depth estimatefor a given pixel location in the image from the reference viewpointbased upon the visibility of the pixels in the set of images from thereference viewpoint by: generating an updated subset of the set ofimages using images in which the given pixel location in the image fromthe reference viewpoint is determined to be visible based upon thecurrent depth estimate for the given pixel; identifying pixels in theupdated subset of the set of images that correspond to the given pixellocation in the image from the reference viewpoint based upon expecteddisparity at a plurality of depths; comparing the similarity of thecorresponding pixels in the updated subset of images identified at eachof the plurality of depths; and selecting the depth from the pluralityof depths at which the identified corresponding pixels in the updatedsubset of the set of images have the highest degree of similarity as anupdated depth estimate for the given pixel location in the image fromthe reference viewpoint.

In yet another embodiment again, the subsets of the set of images arepairs of images; and the updated subset of the set of images includes atleast three images.

In a still further additional embodiment, normalizing the set of imagesto increase the similarity of corresponding pixels within the set ofimages further includes utilizing calibration information to correct forphotometric variations and scene-independent geometric distortions inthe images in the set of images, and rectification of the images in theset of images

In still another additional embodiment, normalizing the set of images toincrease the similarity of corresponding pixels within the set of imagesfurther includes resampling the images to increase the similarity ofcorresponding pixels in the set of images; and the scene-independentgeometric corrections applied to the images are determined at asub-pixel resolution.

In a yet further additional embodiment, utilizing calibrationinformation to correct for photometric variations further includesperforming any one of the normalization processes selected from thegroup consisting of: Black Level calculation and adjustment; vignettingcorrection; lateral color correction; and temperature normalization.

In yet another additional embodiment, the scene-independent geometriccorrections also include rectification to account for distortion androtation of lenses in an array of cameras that captured the set ofimages.

In a further additional embodiment again, a cost function is utilized todetermine the similarity of corresponding pixels.

In another additional embodiment again, determining the similarity ofcorresponding pixels further includes spatially filtering the calculatedcosts.

In another further embodiment, selecting the depth from the plurality ofdepths at which the identified corresponding pixels have the highestdegree of similarity as an initial depth estimate for the given pixellocation in the image from the reference viewpoint further includesselecting the depth from the plurality of depths at which the filteredcost function for the identified corresponding pixels indicates thehighest level of similarity.

In still another further embodiment, the cost function utilizes at leastone similarity measure selected from the group consisting of: the L1norm of a pair of corresponding pixels; the L2 norm of a pair ofcorresponding pixels; and the variance of a set of corresponding pixels.

In yet another further embodiment, the set of images are captured withinmultiple color channels and the cost function determines the similarityof pixels in each of the multiple color channels.

Another further embodiment again also includes generating confidencemetrics for the current depth estimates for pixel locations in the imagefrom the reference viewpoint.

In another further additional embodiment, the confidence metric encodesa plurality of confidence factors.

Still yet another further embodiment also includes filtering the depthmap based upon the confidence map.

Still another further embodiment again also includes detecting occlusionof pixels in images within the set of images that correspond to specificpixel locations in the image from the reference viewpoint based upon theinitial depth estimates by searching along lines parallel to thebaselines between the reference viewpoint and the viewpoints of theimages in the set of images to locate occluding pixels; when an initialdepth estimate results in the detection of a corresponding pixel in atleast one image being occluded, selecting the current depth estimate forthe pixel location in the image from the reference viewpoint by:determining a set of candidate depth estimates using a plurality ofdifferent subsets of the set of images that exclude the at least oneimage in which the given pixel is occluded; identifying correspondingpixels in each of the plurality of subsets of the set of images basedupon the candidate depth estimates; and selecting the candidate depth ofthe subset having the most similar corresponding pixels as the currentdepth estimate for the pixel location in the image from the referenceviewpoint.

In still another further additional embodiment, searching along linesparallel to the baselines between the reference viewpoint and theviewpoints of the images in the set of images to locate occluding pixelsfurther includes determining that a pixel corresponding to a pixellocation (x₁,y₁) in an image from the reference viewpoint is occluded inan alternate view image by a pixel location (x₂,y₂) in the image fromthe reference viewpoint when

|s ₂ −s ₁−√{square root over ((x ₂ −x ₁)²+(y ₂ −y ₁)²)}|≦threshold

where s₁ and s₂ are scene dependent geometric shifts applied to pixellocations (x₁,y₁) and pixel (x₂,y₂) to shift the pixels along a lineparallel to the baseline between the reference viewpoint and theviewpoint of the alternate view image to shift the pixels into theviewpoint of the alternate view image based upon the initial depthestimates for each pixel.

In yet another further embodiment again, the decision to designate apixel as being occluded considers at least one of the similarity of thepixels and the confidence of the estimated depths of the pixels (x₁,y₁)and (x₂,y₂).

In a specific embodiment, a cost function is utilized to determine thesimilarity of corresponding pixels.

In another specific embodiment, determining the similarity ofcorresponding pixels further comprises spatially filtering thecalculated costs.

In a further specific embodiment, the spatial filtering of thecalculated costs utilizes a filter selected from the group consistingof: a fixed-coefficient filter; and an edge-preserving filter.

In a still further specific embodiment, selecting the depth from theplurality of depths at which the identified corresponding pixels havethe highest degree of similarity as an initial depth estimate for thegiven pixel location in the image from the reference viewpoint furtherincludes selecting the depth from the plurality of depths at which thefiltered cost function for the identified corresponding pixels indicatesthe highest level of similarity.

In still another specific embodiment, the set of images are capturedwithin a single color channel and the cost function is a function of thevariance of the corresponding pixel.

In a yet further specific embodiment, the cost function is an aggregatedcost function CV(x,y,d) over each image i in the set of images thatincludes the following term

${{CV}( {x,y,d} )} = {\sum\limits_{i}\frac{{{Cost}^{i,{Ref}}( {x,y,d} )} \times {V^{i,{Ref}}( {x,y} )}}{{number}\mspace{14mu} {of}\mspace{14mu} {visible}\mspace{14mu} {cameras}\mspace{14mu} {at}\mspace{14mu} ( {x,y} )}}$

where Cost^(i,Ref)(x,y,d) is a similarity measure (i.e. the costfunction),

-   -   d is depth of pixel (x,y), and    -   V^(i,Ref)(x,y) is the visibility of pixel (x,y) and initially        V^(i,Ref)(x,y)=1 for all cameras.

In a further specific embodiment again, the individual costsCost^(i,Ref)(x,y,d) are computed based on each disparity hypothesis dfor each pixel (x,y) for cameras i, Ref as follows:

Cost^(i,Ref)(x,y,d)=S{I ^(i)(x,y,d),I ^(Ref)(x,y,d)}

where S is the similarity measure (for example), and

-   -   I^(i) is the calibrated image i after geometric calibration.

In yet another specific embodiment, the aggregated cost considers thesimilarity of the shifted images at the candidate depth as follows:

${{CV}( {x,y,d} )} = {\Sigma_{k \in K}\frac{( {x,y} ){{Cost}^{k,{Ref}}( {x,y,d} )} \times V^{k,{Ref}}}{{number}\mspace{14mu} {of}\mspace{14mu} {cameras}\mspace{14mu} {in}\mspace{14mu} K} \times \Sigma_{i,{j \in L}}\frac{{{Cost}^{i,j}( {x,y,d} )} \times {V^{i,{Ref}}( {x,y} )} \times {V^{j,{Ref}}( {x,y} )}}{{number}\mspace{14mu} {of}\mspace{14mu} {pairs}\mspace{14mu} {of}\mspace{14mu} {cameras}\mspace{14mu} {in}\mspace{14mu} L}}$

where K is a set of cameras in the same spectral channel as thereference camera,

L is a set of pairs of cameras, where both cameras in each pair are inthe same spectral channel (which can be a different spectral channel tothe reference camera where the light field includes image data inmultiple spectral channels),

Cost^(k,Ref)(x,y,d)=S{ImageRef(x,y),ShiftedImage^(k)(x,y,d)}, and

Cost^(i,j)(x,y,d)=S{ShiftedImage^(i)(x,y,d),ShiftedImage^(j)(x,y,d)}

In a further specific embodiment again, the aggregated cost function isspatially filtered using a filter so that the weighted aggregated costfunction is as follows:

${{FilteredCV}( {x,y,d} )} = {\frac{1}{Norm}{\sum\limits_{{({x_{1},y_{1}})} \in {N{({x,y})}}}{{{CV}( {x_{1},y_{1},d} )} \times {{wd}( {x,y,x_{1},y_{1}} )} \times {{wr}( {{I_{Ref}( {x,y} )} - {I_{Ref}( {x_{1},y_{1}} )}} )}}}}$

where N(x,y) is the immediate neighborhood of the pixel (x,y), which canbe square, circular, rectangular, or any other shape appropriate to therequirements of a specific application,

-   -   Norm is a normalization term,    -   I_(Ref)(x,y) is the image data from the reference camera,    -   wd is a weighting function based on pixel distance, and    -   wr is a weighting function based on intensity difference.

In a further embodiment, the filter is a box filter and wd and wr areconstant coefficients.

In another embodiment, the filter is a bilateral filter and wd and wrare both Gaussian weighting functions.

In a still further embodiment, a depth estimate for a pixel location(x,y) in the image from the reference viewpoint is determined byselecting the depth that minimizes the filtered cost at each pixellocation in the depth map as follows:

D(x,y)=argmin{FilteredCV(x,y,d)}

In still another embodiment, the set of images are captured withinmultiple color channels and the cost function incorporates the L1 normof image data from the multiple color channels.

In a yet further embodiment, the set of images are captured withinmultiple color channels and the cost function incorporates the L2 normof image data from the multiple color channels.

In yet another embodiment, the set of images are captured withinmultiple color channels including at least Red, Green and Blue colorchannels; selecting a reference viewpoint relative to the viewpoints ofthe set of images captured from different viewpoints comprises selectingone of the images in the Green color channel as a Green reference imageand selecting the viewpoint of the Green reference image as thereference viewpoint; and the cost function Cost(x,y,d) for a pixellocation (x,y) in the image from the reference viewpoint at a depth dis:

Cost(x,y,d)=γ_(G)(x,y)·Cost_(G)(x,y,d)+γ_(R)(x,y)·Cost_(R)(x,y,d)+γ_(B)(x,y)·Cost_(B)(x,y,d)

where Cost_(G)(x,y,d) is the measure of the similarity of a pixellocation (x,y) in the image from the reference viewpoint tocorresponding pixels in locations within a set of Green images basedupon the depth d,

Cost_(R)(x,y,d) is the measure of the similarity of corresponding pixelsin locations within a set of Red images determined based upon the depthd and the pixel location (x,y) in the image from the referenceviewpoint,

Cost_(B)(x,y,d) is the measure of the similarity of corresponding pixelsin locations within a set of Blue images determined based upon the depthd and the pixel location (x,y) in the image from the referenceviewpoint, and

γ_(G), γ_(R), and γ_(B) are weighting factors for the Green, Red andBlue cost functions respectively.

In a further embodiment again, the Cost_(G)(x,y,d) uses a similaritymeasure selected from the group consisting of an L1 norm, an L2 norm,and variance across the pixels in the images in the set of images thatare within the Green color channel.

In another embodiment again, the cost measures for the Red(Cost_(R)(x,y,d)) and Blue color channels (Cost_(B)(x,y,d)) aredetermined by calculating the aggregated difference between unique pairsof corresponding pixels in images within the color channel.

In a further additional embodiment, calculating the aggregateddifference between each unique pair of corresponding pixels in imageswithin a color channel comprises determining a combination cost metricfor unique pairs of corresponding pixels in images within the colorchannel.

In another additional embodiment, the combination cost metric(Cost_(C)(x,y,d)) for a Red color channel including four images (C_(A),C_(B), C_(C), and C_(D)) can be determined as follows:

Cost_(C)(x,y,d)=|C _(A)(x _(A) y _(A))−C _(B)(x _(B) ,y _(B))|+|C _(A)(x_(A) ,y _(A))−C _(C)(x _(C) ,y _(C))|+|C _(A)(x _(A) ,y _(A))−C _(D)(x_(D) ,y _(D))|+|C _(B)(x _(B) ,y _(B))−C _(C)(x _(C) ,y _(C))|+|C _(B)(x_(B) ,y _(B))−C _(D)(x _(D) ,y _(D))|+|C _(C)(x _(C) ,y _(C))−C _(D)(x_(D) ,y _(D))|

where (x_(A),y_(A)), (x_(B),y_(B)), (x_(C),y_(C)), and (x_(D),y_(D)) arecorresponding pixel locations determined based upon the disparity ineach of the images C_(A), C_(B), C_(C), and C_(D) respectively at depthd.

In a still yet further embodiment, the combination cost metric isdetermined utilizing at least one selected from the group consisting of:the L1 norm of the pixel brightness values; the L2 norm of the pixelbrightness values; and the variance in the pixel brightness values.

In still yet another embodiment, the weighting factors γ_(G), γ_(R), andγ_(B) are fixed.

In a still further embodiment again, the weighting factors γ_(G), γ_(R),and γ_(B) vary spatially with the pixel location (x,y) in the image fromthe reference viewpoint.

In still another embodiment again, the weighting factors γ_(G), γ_(R),and γ_(B) vary based upon the estimated SNR at the pixel location (x,y)in the image from the reference viewpoint; and strong SNR at the pixellocation (x,y) in the image from the reference viewpoint is used toreduce the weighting applied to the Red and Blue color channels.

In a further embodiment, the confidence metric encodes a plurality ofconfidence factors.

In another embodiment, the confidence metric for the depth estimate fora given pixel location in the image from the reference viewpointcomprises at least one confidence factor selected from the groupconsisting of: an indication that the given pixel is within atextureless region within an image; a measure of the signal to noiseration (SNR) in a region surrounding a given pixel; the number ofcorresponding pixels used to generate the depth estimate; an indicationof the number of depths searched to generate the depth estimate; anindication that the given pixel is adjacent a high contrast edge; and anindication that the given pixel is adjacent a high contrast boundary.

In a still further embodiment, the confidence metric for the depthestimate for a given pixel location in the image from the referenceviewpoint comprises at least one confidence factor selected from thegroup consisting of: an indication that the given pixel lies on agradient edge; an indication that the corresponding pixels to the givenpixel are mismatched; an indication that corresponding pixels to thegiven pixel are occluded; an indication that depth estimates generatedusing different reference cameras exceed a threshold for the givenpixel; an indication that the depth estimates generated using differentsubsets of cameras exceed a threshold for the given pixel; an indicationas to whether the depth of the given threshold exceeds a threshold; anindication that the given pixel is defective; and an indication thatcorresponding pixels to the given pixel are defective.

In still another embodiment, the confidence metric for the depthestimate for a given pixel location in the image from the referenceviewpoint comprises at least: a measure of the SNR in a regionsurrounding a given pixel; and the number of corresponding pixels usedto generate the depth estimate.

In a yet further embodiment, the confidence metric encodes at least onebinary confidence factor.

In yet another embodiment, the confidence metric encodes at least oneconfidence factor represented as a range of degrees of confidence.

In a further embodiment again, the confidence metric encodes at leastone confidence factor determined by comparing the similarity of thepixels in the set of images that were used to generate the finalizeddepth estimate for a given pixel location in the image from thereference viewpoint.

In another embodiment again, a cost function is utilized to generate acost metric indicating the similarity of corresponding pixels; andcomparing the similarity of the pixels in the set of images that wereused to generate the depth estimate for a given pixel location in theimage from the reference viewpoint further comprises: applying athreshold to a cost metric of the pixels in the set of images that wereused to generate the finalized depth estimate for a given pixel locationin the image from the reference viewpoint; and when the cost metricexceeds the threshold, assigning a confidence metric that indicates thatthe finalized depth estimate for the given pixel location in the imagefrom the reference viewpoint was generated using at least one pixel inthe set of images that is a problem pixel.

In a further additional embodiment, the threshold is modified based uponat least one of: a mean intensity of a region surrounding the givenpixel location in the image from the reference viewpoint; and noisestatistics for at least one sensor used to capture the set of images.

In a still yet further embodiment, the mean intensity of a regionsurrounding the given pixel location in the image from the referenceviewpoint is calculated using a spatial box N×N averaging filtercentered around the given pixel.

In still yet another embodiment, the set of images are captured withinmultiple color channels including at least Red, Green and Blue colorchannels; selecting a reference viewpoint relative to the viewpoints ofthe set of images captured from different viewpoints comprises selectingone of the images in the Green color channel as a Green reference imageand selecting the viewpoint of the Green reference image as thereference viewpoint; and the mean intensity is used to determine thenoise statistics for the Green channel using a table that relates aparticular mean at a particular exposure and gain to a desiredthreshold.

In a still further embodiment again, selecting a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints comprises selecting one of the images as a reference imageand selecting the viewpoint of the reference image as the referenceviewpoint; and a cost function is utilized to generate a cost metricindicating the similarity of corresponding pixels; a confidence metricbased upon general mismatch is obtained using the following formula:

Confidence(x,y)=F(Cost_(min)(x,y),Cost^(d)(x,y),I(x,y)^(cam),Sensor)

where Cost_(min)(x,y) is the minimum cost of a disparity search over thedesired depth range,

-   -   Cost^(d)(x,y) denotes that cost data from any depth or depths        (beside the minimum depth),    -   I(x,y)^(cam) image data captured by any camera can be utilized        to augment the confidence;    -   Sensor is the sensor prior, which can include known properties        of the sensor, such as (but not limited to) noise statistics or        characterization, defective pixels, properties of the sensor        affecting any captured images (such as gain or exposure),    -   Camera intrinsics is the camera intrinsic, which specifies        elements intrinsic to the camera and camera array that can        impact confidence including (but not limited to) the baseline        separation between cameras in the array (affects precision of        depth measurements), and the arrangement of the color filters        (affects performance in the occlusion zones in certain        scenarios).

In still another embodiment again, selecting a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints comprises selecting one of the images as a reference imageand selecting the viewpoint of the reference image as the referenceviewpoint; and a cost function is utilized to generate a cost metricindicating the similarity of corresponding pixels; and a confidencemetric based upon general mismatch is obtained using the followingformula:

${{Confidence}( {x,y} )} = {{a \times \frac{{Cost}_{\min}( {x,y} )}{{Avg}( {x,y} )}} + {offset}}$

where Avg(x,y) is the mean intensity of the reference image in a spatialneighborhood surrounding (x,y), or an estimate of the mean intensity inthe neighborhood, that is used to adjust the confidence based upon theintensity of the reference image in the region of (x,y),

-   -   a and offset are empirically chosen scale and offset factors        used to adjust the confidence with prior information about the        gain and noise statistics of the sensor.    -   a and offset are empirically chosen scale and offset factors        used to adjust the confidence with prior information about the        gain and noise statistics of at least one sensor used to capture        images in the set of images.

In a yet further embodiment again, generating confidence metrics for thedepth estimates for pixel locations in the image from the referenceviewpoint includes determining at least one sensor gain used to captureat least one of the set of images and adjusting the confidence metricsbased upon the sensor gain.

In yet another embodiment again, generating confidence metrics for thedepth estimates for pixel locations in the image from the referenceviewpoint comprises determining at least one exposure time used tocapture at least one of the set of images and adjusting the confidencemetrics based upon the sensor gain.

A still further additional embodiment also includes outputting a depthmap containing the finalized depth estimates for pixel locations in theimage from the reference viewpoint, and outputting a confidence mapcontaining confidence metrics for the finalized depth estimatescontained within the depth map.

Still another additional embodiment also includes filtering the depthmap based upon the confidence map.

Yet another further additional embodiment includes estimating distancesto objects within a scene from the light field comprising a set ofimages captured from different viewpoints using a processor configuredby an image processing application by: selecting a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints; normalizing the set of images to increase the similarity ofcorresponding pixels within the set of images; determining initial depthestimates for pixel locations in an image from the reference viewpointusing at least a subset of the set of images, where an initial depthestimate for a given pixel location in the image from the referenceviewpoint is determined by: identifying pixels in the at least a subsetof the set of images that correspond to the given pixel location in theimage from the reference viewpoint based upon expected disparity at aplurality of depths; comparing the similarity of the correspondingpixels identified at each of the plurality of depths; and selecting thedepth from the plurality of depths at which the identified correspondingpixels have the highest degree of similarity as an initial depthestimate for the given pixel location in the image from the referenceviewpoint. In addition, the process of estimating distances furtherincludes identifying corresponding pixels in the set of images using theinitial depth estimates; comparing the similarity of the correspondingpixels in the set of images to detect mismatched pixels; when an initialdepth estimate does not result in the detection of a mismatch betweencorresponding pixels in the set of images, selecting the initial depthestimate as the current depth estimate for the pixel location in theimage from the reference viewpoint; and when an initial depth estimateresults in the detection of a mismatch between corresponding pixels inthe set of images, selecting the current depth estimate for the pixellocation in the image from the reference viewpoint by: determining a setof candidate depth estimates using a plurality of different subsets ofthe set of images; identifying corresponding pixels in each of theplurality of subsets of the set of images based upon the candidate depthestimates; and selecting the candidate depth of the subset having themost similar corresponding pixels as the current depth estimate for thepixel location in the image from the reference viewpoint. The processfurther including determining the visibility of the pixels in the set ofimages from the reference viewpoint by: identifying corresponding pixelsin the set of images using the current depth estimates; and determiningthat a pixel in a given image is not visible in the reference viewpointwhen the pixel fails a photometric similarity criterion determined basedupon a comparison of corresponding pixels; and fusing pixels from theset of images using the processor configured by the image processingapplication based upon the depth estimates to create a fused imagehaving a resolution that is greater than the resolutions of the imagesin the set of images by: identifying the pixels from the set of imagesthat are visible in an image from the reference viewpoint using thevisibility information; and applying scene dependent geometric shifts tothe pixels from the set of images that are visible in an image from thereference viewpoint to shift the pixels into the reference viewpoint,where the scene dependent geometric shifts are determined using thecurrent depth estimates; and fusing the shifted pixels from the set ofimages to create a fused image from the reference viewpoint having aresolution that is greater than the resolutions of the images in the setof images.

Another further embodiment also includes synthesizing an image from thereference viewpoint using the processor configured by the imageprocessing application to perform a super resolution process based uponthe fused image from the reference viewpoint, the set of images capturedfrom different viewpoints, the current depth estimates, and thevisibility information.

A further embodiment of the invention includes a processor, and memorycontaining a set of images captured from different viewpoints and animage processing application. In addition, the image processingapplication configures the processor to: select a reference viewpointrelative to the viewpoints of the set of images captured from differentviewpoints; normalize the set of images to increase the similarity ofcorresponding pixels within the set of images; determine initial depthestimates for pixel locations in an image from the reference viewpointusing at least a subset of the set of images, where an initial depthestimate for a given pixel location in the image from the referenceviewpoint is determined by: identifying pixels in the at least a subsetof the set of images that correspond to the given pixel location in theimage from the reference viewpoint based upon expected disparity at aplurality of depths; comparing the similarity of the correspondingpixels identified at each of the plurality of depths; and selecting thedepth from the plurality of depths at which the identified correspondingpixels have the highest degree of similarity as an initial depthestimate for the given pixel location in the image from the referenceviewpoint. The application further configures the processor to identifycorresponding pixels in the set of images using the initial depthestimates; compare the similarity of the corresponding pixels in the setof images to detect mismatched pixels. When an initial depth estimatedoes not result in the detection of a mismatch between correspondingpixels in the set of images, the application configures the processor toselect the initial depth estimate as the current depth estimate for thepixel location in the image from the reference viewpoint. When aninitial depth estimate results in the detection of a mismatch betweencorresponding pixels in the set of images, the application configuresthe processor to select the current depth estimate for the pixellocation in the image from the reference viewpoint by: determining a setof candidate depth estimates using a plurality of different subsets ofthe set of images; identifying corresponding pixels in each of theplurality of subsets of the set of images based upon the candidate depthestimates; and selecting the candidate depth of the subset having themost similar corresponding pixels as the current depth estimate for thepixel location in the image from the reference viewpoint.

In another embodiment, the image processing application furtherconfigures the processor to: determine the visibility of the pixels inthe set of images from the reference viewpoint by: identifyingcorresponding pixels in the set of images using the current depthestimates; and determining that a pixel in a given image is not visiblein the reference viewpoint when the pixel fails a photometric similaritycriterion determined based upon a comparison of corresponding pixels;and fuse pixels from the set of images using the depth estimates tocreate a fused image having a resolution that is greater than theresolutions of the images in the set of images by: identifying thepixels from the set of images that are visible in an image from thereference viewpoint using the visibility information; and applying scenedependent geometric shifts to the pixels from the set of images that arevisible in an image from the reference viewpoint to shift the pixelsinto the reference viewpoint, where the scene dependent geometric shiftsare determined using the current depth estimates; and fusing the shiftedpixels from the set of images to create a fused image from the referenceviewpoint having a resolution that is greater than the resolutions ofthe images in the set of images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 conceptual illustrates of an array camera in accordance with anembodiment of the invention.

FIG. 1A conceptually illustrates an array camera module in accordancewith an embodiment of the invention.

FIG. 1C conceptually illustrates a color filter pattern for a 4×4 arraycamera module in accordance with an embodiment of the invention.

FIG. 2 conceptually illustrates capturing image data using a referencecamera and an alternate view camera.

FIGS. 3A and 3B conceptually illustrate the effect of parallax in imagesof a scene captured by a reference camera and an alternate view camera.

FIG. 4 is a flowchart illustrating a process for generating a depth mapfrom a captured light field including a plurality of images capturedfrom different viewpoints in accordance with an embodiment of theinvention.

FIG. 5 is a flowchart of a process for normalizing captured image datain accordance with an embodiment of the invention.

FIG. 6 is a flowchart of a process for iteratively refining a depth mapbased upon visibility information in accordance with embodiments of theinvention.

FIG. 7 conceptually illustrates a subset of cameras within an arraycamera that can be utilized to generate estimates of distances toobjects within a scene in accordance with an embodiment of theinvention.

FIG. 8 is a flowchart illustrating a process for performing a disparitysearch using visibility information in accordance with an embodiment ofthe invention.

FIG. 8A is a flowchart illustrating a process for estimating depth usingimages captured by subsets of cameras in a camera array in accordancewith an embodiment of the invention.

FIGS. 8B-8I conceptually illustrate subsets of cameras in a 5×5 arraycamera that can be utilized to obtain depth estimates in accordance withembodiments of the invention.

FIGS. 8J-8M conceptually illustrate subsets of cameras in a 4×4 arraycamera that can be utilized to obtain depth estimates in accordance withembodiments of the invention.

FIG. 9 conceptually illustrates a process for searching an epipolar linefor pixels that occlude a given pixel in accordance with an embodimentof the invention.

FIG. 10 conceptually illustrates a 5×5 array camera that can be utilizedto construct a depth map in accordance with an embodiment of theinvention.

FIG. 11 is a flowchart illustrating a process for determining visibilitybased upon the photometric similarity of corresponding pixels inaccordance with an embodiment of the invention.

FIG. 12 conceptually illustrates one of many virtual viewpoints that canbe defined with respect to a 4×4 array camera in accordance with anembodiment of the invention.

FIG. 13 is a flowchart illustrating a process for generating a sparsedepth map in accordance with an embodiment of the invention.

FIG. 14 conceptually illustrates a set of pixels that can be utilized asindicator pixels when generating a sparse depth map in accordance withan embodiment of the invention.

FIG. 15 is a flowchart illustrating a process for detecting texturelessregions using the SNR surrounding a pixel in accordance with anembodiment of the invention.

FIG. 16 is a system for generating a depth map and visibilityinformation in accordance with an embodiment of the invention.

FIG. 17 is a flowchart illustrating a process for synthesizing a higherresolution image from a plurality of lower resolution images capturedfrom different viewpoints using super-resolution processing inaccordance with an embodiment of the invention.

FIGS. 18A and 18B conceptually illustrate sources of noise in depthestimates.

FIGS. 18C-18H conceptually illustrate the generation of a depth map anda confidence map from captured image data and the use of the confidencemap to filter the depth map in accordance with an embodiment of theinvention.

FIGS. 18I-18N similarly conceptually illustrate the generation of adepth map and a confidence map from captured image data and the use ofthe confidence map to filter the depth map using close up images inaccordance with an embodiment of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for parallax detectionand correction in images captured using array cameras are illustrated.Array cameras, such as those described in U.S. patent application Ser.No. 12/935,504 entitled “Capturing and Processing of Images usingMonolithic Camera Array with Heterogeneous Imagers” to Venkataraman etal., can be utilized to capture light field images. In a number ofembodiments, super-resolution processes such as those described in U.S.patent application Ser. No. 12/967,807 entitled “Systems and Methods forSynthesizing High Resolution Images Using Super-Resolution Processes” toLelescu et al., are utilized to synthesize a higher resolution 2D imageor a stereo pair of higher resolution 2D images from the lowerresolution images in the light field captured by an array camera. Theterms high or higher resolution and low or lower resolution are usedhere in a relative sense and not to indicate the specific resolutions ofthe images captured by the array camera. The disclosures of U.S. patentapplication Ser. No. 12/935,504 and U.S. patent application Ser. No.12/967,807 are hereby incorporated by reference in their entirety.

Each two-dimensional (2D) image in a captured light field is from theviewpoint of one of the cameras in the array camera. Due to thedifferent viewpoint of each of the cameras, parallax results invariations in the position of objects within the different images of thescene. Systems and methods in accordance with embodiments of theinvention provide an accurate account of the pixel disparity as a resultof parallax between the different cameras in the array, so thatappropriate scene-dependent geometric shifts can be applied to thepixels of the captured images when performing super-resolutionprocessing.

A high resolution image synthesized using super-resolution processing issynthesized from a specific viewpoint that can be referred to as areference viewpoint. The reference viewpoint can be from the viewpointof one of the cameras in a camera array. Alternatively, the referenceviewpoint can be an arbitrary virtual viewpoint where there is nophysical camera. A benefit of synthesizing a high resolution image fromthe viewpoint of one of the cameras (as opposed to a virtual viewpoint)is that the disparity of the pixels in the light field can be determinedwith respect to the image in the light field captured from the referenceviewpoint. When a virtual viewpoint is utilized, none of the capturedimage data is from the reference viewpoint and so the process insteadrelies solely on cameras away from the reference position to determinethe best match.

Array cameras in accordance with many embodiments of the invention usethe disparity between the pixels in the images within a light field togenerate a depth map from the reference viewpoint. A depth map indicatesthe distance of scene objects from a reference viewpoint and can beutilized to determine scene dependent geometric corrections to apply tothe pixels from each of the images within a captured light field tocorrect for disparity when performing super-resolution processing. Inseveral embodiments, an initial depth map of the reference viewpoint isgenerated and as part of that process or as a subsequent processoccluded pixels and/or other types of mismatched pixels are detected.The process of detecting pixels that are occluded can also be thought ofas determining whether a pixel in an image captured from the referenceviewpoint is visible in the image from a non-reference viewpoint. When apixel in the image captured from the reference viewpoint is not visiblein a second image, utilizing image data from the second image whendetermining the depth of the pixel in the reference image introduceserror into the depth determination. Therefore, by detecting the pixelsin the reference image that are occluded in one or more images in thelight field, the accuracy of the depth map can be improved. In severalembodiments, the initial depth map is updated by determining the depthsof occluded pixels using image data captured from cameras in which thepixels are visible (i.e. not occluded). In a number of embodiments, thelikely presence of occlusions and/or other sources of mismatched pixelscan be detected during the process of generating an initial depthestimate and subsets of a set of images that correspond to differentpatterns of visibility within a scene can be used to determine a set ofcandidate depth estimates. The candidate depth of the subset of imageshaving the most similar corresponding pixels can be used as the newdepth estimate and the new depth estimate used to determine thevisibility of the corresponding pixels in some or all of the remainingset of images.

A depth map from a reference viewpoint can be utilized to determine thescene dependent geometric shifts that are likely to have occurred inimages captured from other viewpoints. These scene dependent geometricshifts can be utilized in super-resolution processing. In addition, thescene dependent geometric shifts can be utilized to refine thedeterminations of the visibility of pixels within the light field fromthe reference viewpoint. In a number of embodiments, the scene dependentgeometric shifts are utilized to compare the similarity of pixels.Assuming the depth of a pixel from the reference viewpoint is correctlydetermined, then the similarity of the pixels is indicative of whetherthe pixel is visible. A similar pixel is likely to be the pixel observedfrom the reference viewpoint shifted due to disparity. If the pixels aredissimilar, then the pixel observed from the reference viewpoint islikely occluded in the second image. In many embodiments, visibilityinformation is utilized in further updating depth maps. In severalembodiments, visibility information is generated and provided along withthe depth map for use in super-resolution processing.

In a number of embodiments, the computational complexity of generatingdepth maps is reduced by generating a sparse depth map that includesadditional depth estimates in regions where additional depth informationis desirable such as (but not limited to) regions involving depthtransitions and/or regions containing pixels that are occluded in one ormore images within the light field.

Many array cameras capture color information using different cameras(see for example the array cameras disclosed in U.S. patent applicationSer. No. 12/935,504). In many embodiments, the viewpoint of a Greencamera is utilized as the reference viewpoint. An initial depth map canbe generated using the images captured by other Green cameras in thearray camera and the depth map used to determine the visibility of Red,Green, and Blue pixels within the light field. In other embodiments,image data in multiple color channels can be utilized to perform depthestimation. In several embodiments, the similarity of correspondingpixels in each color channel is considered when estimating depth. In anumber of embodiments, the similarity of sets of corresponding pixels indifferent color channels is also considered when estimating depth. Depthestimation using various cost functions that consider the similarity ofcorresponding pixels at specific depths in a single spectral channel, inmultiple spectral channels, and/or across spectral channels inaccordance with embodiments of the invention are discussed furtherbelow.

In several embodiments, the array camera can include one or more camerasthat capture image data in multiple color channels. For example, anarray camera may include one or more cameras that have a Bayer colorfilter pattern, in addition to or as an alternative to monochromecameras. When the viewpoint of a camera that captures multiple colorchannels is used as the reference viewpoint for the purpose ofgenerating a depth map, a depth map and visibility information can bedetermined for each color channel captured from the reference viewpoint.When a reference image contains information concerning multiple colorchannels, depth and visibility information can be more reliably createdbased upon the disparity of the pixels in the light field with respectto the reference image than by registering the pixels in one channelwith respect to the depth and visibility of pixels in another colorchannel. A disadvantage of utilizing the viewpoint of a camera thatcaptures image data in multiple color channels as a reference viewpointis that the resolution of the depth information in each of the capturedcolor channels is reduced relative to a camera that captures image datausing the same number of pixels in a single channel. Accordingly, theconfiguration of the array camera and the selection of the viewpoint toutilize as the reference viewpoint typically depend upon therequirements of a specific application.

Once a depth map and visibility information are generated for the pixelsin the light field, the depth map and visibility information can beprovided to a super-resolution processing pipeline in accordance withembodiments of the invention to synthesize a higher resolution 2D imageof the scene. The depth map can be utilized to correct for parallaxbetween the different low resolution images and visibility informationcan be utilized during fusion to prevent the fusion of occluded pixels(i.e. pixels in an alternate view image that are not visible from thereference viewpoint). In several embodiments, the process of generatinga depth map also includes generating a confidence map that includesconfidence metrics for the depth estimates in the depth map. In severalembodiments, the depth metrics encode at least one confidence factorindicative of the reliability of the corresponding depth estimate. In anumber of embodiments, the confidence metric includes at least aconfidence factor based on the signal to noise ratio (SNR) in the regionof the pixel location with which the depth estimate is associated, and aconfidence factor based upon the number of pixels in a set of imagesthat correspond to the pixel location with which the depth map isassociated that were utilized to generate the depth estimate and/or areoccluded. Systems and methods for detecting and correcting disparity inimages captured by array cameras in accordance with embodiments of theinvention are described below. Before discussing the detection andcorrection of parallax, however, various array cameras in accordancewith embodiments of the invention are discussed.

Array Camera Architecture

Array cameras in accordance with embodiments of the invention caninclude a camera module including an array of cameras and a processorconfigured to read out and process image data from the camera module tosynthesize images. An array camera in accordance with an embodiment ofthe invention is illustrated in FIG. 1. The array camera 100 includes acamera module 102 with an array of individual cameras 104 where an arrayof individual cameras refers to a plurality of cameras in a particulararrangement, such as (but not limited to) the square arrangementutilized in the illustrated embodiment. The camera module 102 isconnected to the processor 108. The processor is also configured tocommunicate with one or more different types of memory 110 that can beutilized to store image data and/or contain machine readableinstructions utilized to configure the processor to perform processesincluding (but not limited to) the various processes described below. Inmany embodiments, the memory contains an image processing applicationthat is configured to process a light field comprising a plurality ofimages to generate a depth map(s), a visibility map(s), a confidencemap(s), and/or a higher resolution image(s) using any of the processesoutlined in detail below. As is discussed further below, a depth maptypically provides depth estimates for pixels in an image from areference viewpoint (e.g. a higher resolution image synthesized from areference viewpoint). A variety of visibility maps can be generated asappropriate to the requirements of specific applications including (butnot limited to) visibility maps indicating whether pixel locations in areference image are visible in specific images within a light field,visibility maps indicating whether specific pixels in an image withinthe light field are visible from the reference viewpoint, and visibilitymaps indicating whether a pixel visible in one alternate view image isvisible in another alternate view image. In other embodiments, any of avariety of applications can be stored in memory and utilized to processimage data using the processes described herein. In several embodiments,processes in accordance with embodiments of the invention can beimplemented in hardware using an application specific integrationcircuit, and/or a field programmable gate array, or implementedpartially in hardware and software.

Processors 108 in accordance with many embodiments of the invention areconfigured using appropriate software to take the image data within thelight field and synthesize one or more high resolution images. Inseveral embodiments, the high resolution image is synthesized from areference viewpoint, typically that of a reference focal plane 104within the sensor 102. In many embodiments, the processor is able tosynthesize an image from a virtual viewpoint, which does not correspondto the viewpoints of any of the focal planes 104 in the sensor 102. Theimages in the light field will include a scene-dependent disparity dueto the different fields of view of the focal planes used to capture theimages. Processes for detecting and correcting for disparity arediscussed further below. Although a specific array camera architectureis illustrated in FIG. 1, alternative architectures can also be utilizedin accordance with embodiments of the invention.

Array Camera Modules

Array camera modules in accordance with embodiments of the invention canbe constructed from an imager array or sensor including an array offocal planes and an optic array including a lens stack for each focalplane in the imager array. Sensors including multiple focal planes arediscussed in U.S. patent application Ser. No. 13/106,797 entitled“Architectures for System on Chip Array Cameras”, to Pain et al., thedisclosure of which is incorporated herein by reference in its entirety.Light filters can be used within each optical channel formed by the lensstacks in the optic array to enable different cameras within an arraycamera module to capture image data with respect to different portionsof the electromagnetic spectrum (i.e. within different spectralchannels).

An array camera module in accordance with an embodiment of the inventionis illustrated in FIG. 1A. The array camera module 150 includes animager array 152 including an array of focal planes 154 along with acorresponding optic array 156 including an array of lens stacks 158.Within the array of lens stacks, each lens stack 158 creates an opticalchannel that forms an image of the scene on an array of light sensitivepixels within a corresponding focal plane 154. Each pairing of a lensstack 158 and focal plane 154 forms a single camera 104 within thecamera module. Each pixel within a focal plane 154 of a camera 104generates image data that can be sent from the camera 104 to theprocessor 108. In many embodiments, the lens stack within each opticalchannel is configured so that pixels of each focal plane 158 sample thesame object space or region within the scene. In several embodiments,the lens stacks are configured so that the pixels that sample the sameobject space do so with sub-pixel offsets to provide sampling diversitythat can be utilized to recover increased resolution through the use ofsuper-resolution processes. The term sampling diversity refers to thefact that the images from different viewpoints sample the same object inthe scene but with slight sub-pixel offsets. By processing the imageswith sub-pixel precision, additional information encoded due to thesub-pixel offsets can be recovered when compared to simply sampling theobject space with a single image.

In the illustrated embodiment, the focal planes are configured in a 5×5array. Each focal plane 154 on the sensor is capable of capturing animage of the scene. Typically, each focal plane includes a plurality ofrows of pixels that also forms a plurality of columns of pixels, andeach focal plane is contained within a region of the imager that doesnot contain pixels from another focal plane. In many embodiments, imagedata capture and readout of each focal plane can be independentlycontrolled. In this way, image capture settings including (but notlimited to) the exposure times and analog gains of pixels within a focalplane can be determined independently to enable image capture settingsto be tailored based upon factors including (but not limited to) aspecific color channel and/or a specific portion of the scene dynamicrange. The sensor elements utilized in the focal planes can beindividual light sensing elements such as, but not limited to,traditional CIS (CMOS Image Sensor) pixels, CCD (charge-coupled device)pixels, high dynamic range sensor elements, multispectral sensorelements and/or any other structure configured to generate an electricalsignal indicative of light incident on the structure. In manyembodiments, the sensor elements of each focal plane have similarphysical properties and receive light via the same optical channel andcolor filter (where present). In other embodiments, the sensor elementshave different characteristics and, in many instances, thecharacteristics of the sensor elements are related to the color filterapplied to each sensor element.

In several embodiments, color filters in individual cameras can be usedto pattern the camera module with π filter groups as further discussedin U.S. Provisional Patent Application No. 61/641,165 entitled “CameraModules Patterned with pi Filter Groups” filed May 1, 2012, thedisclosure of which is incorporated by reference herein in its entirety.These cameras can be used to capture data with respect to differentcolors, or a specific portion of the spectrum. In contrast to applyingcolor filters to the pixels of the camera, color filters in manyembodiments of the invention are included in the lens stack. Any of avariety of color filter configurations can be utilized including theconfiguration in FIG. 1C including eight Green cameras, four Bluecameras, and four Red cameras, where the cameras are more evenlydistributed around the center of the camera. For example, a Green colorcamera can include a lens stack with a Green light filter that allowsGreen light to pass through the optical channel. In many embodiments,the pixels in each focal plane are the same and the light informationcaptured by the pixels is differentiated by the color filters in thecorresponding lens stack for each filter plane. Although a specificconstruction of a camera module with an optic array including colorfilters in the lens stacks is described above, camera modules includingπ filter groups can be implemented in a variety of ways including (butnot limited to) by applying color filters to the pixels of the focalplanes of the camera module similar to the manner in which color filtersare applied to the pixels of a conventional color camera. In severalembodiments, at least one of the cameras in the camera module caninclude uniform color filters applied to the pixels in its focal plane.In many embodiments, a Bayer filter pattern is applied to the pixels ofone of the cameras in a camera module. In a number of embodiments,camera modules are constructed in which color filters are utilized inboth the lens stacks and on the pixels of the imager array.

Although specific array cameras and imager arrays are discussed above,many different array cameras can be utilized to capture image data andsynthesize images in accordance with embodiments of the invention.Systems and methods for detecting and correcting parallax in image datacaptured by an array camera in accordance with embodiments of theinvention are discussed below.

Determining Parallax/Disparity

In a number of embodiments, the individual cameras in the array cameraused to capture the light field have similar fields of view, fixedapertures, and focal lengths. As a result, the cameras tend to have verysimilar depth of field. Parallax in a two camera system is illustratedin FIG. 2. The two cameras 200, 202, include a lens stack 204 and afocal plane 206. Each camera has a back focal length f, and the twocameras are separated by the baseline distance of 2h. The field of viewof both cameras encompasses a scene including a foreground object 208and a background object 210. The scene from the viewpoint of the firstcamera 200 is illustrated in FIG. 3A. In the image 300 captured by thefirst camera, the foreground object 208 appears located slightly to theright of the background object 210. The scene from the viewpoint of thesecond camera 202 is illustrated in FIG. 3B. In the image 302 capturedby the second camera, the foreground object 208 appears shifted to theleft hand side of the background object 210. The disparity introduced bythe different fields of view of the two cameras 200, 202, is equal tothe difference between the location of the foreground object 208 in theimage captured by the first camera (indicated in the image captured bythe second camera by ghost lines 304) and its location in the imagecaptured by the second camera. As is discussed further below, thedistance from the two cameras to the foreground object can be obtainedby determining the disparity of the foreground object in the twocaptured images.

Referring again to FIG. 2, the point (x_(o),y_(o),z_(o)) on theforeground object will appear on the focal plane of each camera at anoffset from the camera's optical axis. The offset of the point on thefocal plane of the first camera 200 relative to its optical axis 212 isshown as −u_(k). The offset of the point on the focal plane of thesecond camera 202 relative to its optical axis 214 is shown as u_(R).Using similar triangles, the offset between the images captured by thetwo cameras can be observed as follows:

$\frac{h - x_{o}}{z_{o}} = \frac{- u_{L}}{f}$$\frac{h + x_{o}}{z_{o}} = \frac{u_{R}}{f}$

Combining the two equations yields the disparity (or parallax) betweenthe two cameras as:

$\Delta_{parallax} = {{u_{R} - u_{L}} = \frac{2{hf}}{z_{o}}}$

From the above equation, it can be seen that disparity between imagescaptured by the cameras is along a vector in the direction of thebaseline of the two cameras, which can be referred to as the epipolarline between the two cameras. Furthermore, the magnitude of thedisparity is directly proportional to the baseline separation of the twocameras and the back focal length of the cameras and is inverselyproportional to the distance from the camera to an object appearing inthe scene.

Occlusions in Array Cameras

When multiple images of a scene are captured from different perspectivesand the scene includes foreground objects, the disparity in the locationof the foreground object in each of the images results in portions ofthe scene behind the foreground object being visible in some but not allof the images. A pixel that captures image data concerning a portion ofa scene, which is not visible in images captured of the scene from otherviewpoints, can be referred to as an occluded pixel. Referring again toFIGS. 3A and 3B, when the viewpoint of the second camera is selected asa reference viewpoint the pixels contained within the ghost lines 304 inthe image 302 can be considered to be occluded pixels (i.e. the pixelscapture image data from a portion of the scene that is visible in theimage 302 captured by the second camera 202 and is not visible in theimage 300 captured by the first camera 200). The pixels contained in theghost line 306 in the first image can be considered to be revealedpixels (i.e. pixels that are not visible in the reference viewpoint, butthat are revealed by shifting to an alternate viewpoint). In the secondimage, the pixels of the foreground object 208 can be referred to asoccluding pixels as they capture portions of the scene that occlude thepixels contained within the ghost lines 304 in the image 302. Due to theocclusion of the pixels contained within the ghost lines 304 in thesecond image 302, the distance from the camera to portions of the scenevisible within the ghost lines 304 cannot be determined from the twoimages as there are no corresponding pixels in the image 300 shown inFIG. 3A.

As is discussed further below, increasing the number of camerascapturing images of a scene from different viewpoints in complementaryocclusion zones around the reference viewpoint increases the likelihoodthat every portion of the scene visible from the reference viewpoint isalso visible from the viewpoint of at least one of the other cameras.When the array camera uses different cameras to capture differentwavelengths of light (e.g. RGB), distributing at least one camera thatcaptures each wavelength of light in the quadrants surrounding areference viewpoint can significantly decrease the likelihood that aportion of the scene visible from the reference viewpoint will beoccluded in every other image captured within a specific color channel.The distribution of color filters in array cameras to reduce thelikelihood of occlusions in accordance with embodiments of the inventionis discussed further in U.S. Provisional Patent Application Ser. No.61/641,164 entitled “Camera Modules Patterned with π Filter Groups”, toNisenzon et al., filed May 1, 2012, the disclosure of which isincorporated herein by reference in its entirety.

Using Disparity to Generate Depth Maps in Array Cameras

Array cameras in accordance with many embodiments of the invention usedisparity observed in images captured by the array cameras to generate adepth map. A depth map is typically regarded as being a layer ofmetadata concerning an image that describes the distance from the camerato specific pixels or groups of pixels within the image (depending uponthe resolution of the depth map relative to the resolution of theoriginal input images). Array cameras in accordance with a number ofembodiments of the invention use depth maps for a variety of purposesincluding (but not limited to) generating scene dependent geometricshifts during the synthesis of a high resolution image and/or performingdynamic refocusing of a synthesized image.

Based upon the discussion of disparity above, the process of determiningthe depth of a portion of a scene based upon pixel disparity istheoretically straightforward. When the viewpoint of a specific camerain the array camera is chosen as a reference viewpoint, the distance toa portion of the scene visible from the reference viewpoint can bedetermined using the disparity between the corresponding pixels in someor all of the images captured by the camera array. In the absence ofocclusions, a pixel corresponding to a pixel in the image captured fromthe reference viewpoint will be located in each non-reference oralternate view image along an epipolar line (i.e. a line parallel to thebaseline vector between the two cameras). The distance along theepipolar line of the disparity corresponds to the distance between thecamera and the portion of the scene captured by the pixels. Therefore,by comparing the pixels in the captured images that are expected tocorrespond at a specific depth, a search can be conducted for the depththat yields the pixels having the highest degree of similarity. Thedepth at which the corresponding pixels in the captured images have thehighest degree of similarity can be selected as the most likely distancebetween the camera and the portion of the scene captured by the pixel.As is discussed below, similarity can be determined with respect tocorresponding pixels within a single spectral channel, within multiplespectral channels, and/or across spectral channels as appropriate to therequirements of specific applications in accordance with embodiments ofthe invention.

Many challenges exist, however, in determining an accurate depth mapusing the method outlined above. In several embodiments, the cameras inan array camera are similar but not the same. Therefore, imagecharacteristics including (but not limited to) optical characteristics,different sensor characteristics (such as variations in sensor responsedue to offsets, different transmission or gain responses, non-linearcharacteristics of pixel response), noise in the captured images, and/orwarps or distortions related to manufacturing tolerances related to theassembly process can vary between the images reducing the similarity ofcorresponding pixels in different images. In addition, super-resolutionprocesses rely on sampling diversity in the images captured by an imagerarray in order to synthesize higher resolution images. However,increasing sampling diversity can also involve decreasing similaritybetween corresponding pixels in captured images in a light field. Giventhat the process for determining depth outlined above relies upon thesimilarity of pixels, the presence of photometric differences andsampling diversity between the captured images can reduce the accuracywith which a depth map can be determined.

The generation of a depth map is further complicated by occlusions. Asdiscussed above, an occlusion occurs when a pixel that is visible fromthe reference viewpoint is not visible in one or more of the capturedimages. The effect of an occlusion is that at the correct depth, thepixel location that would otherwise be occupied by a corresponding pixelis occupied by a pixel sampling another portion of the scene (typicallyan object closer to the camera). The occluding pixel is often verydifferent to the occluded pixel. Therefore, a comparison of thesimilarity of the pixels at the correct depth is less likely to resultin a significantly higher degree of similarity than at other depths.Effectively, the occluding pixel acts as a strong outlier masking thesimilarity of those pixels, which in fact correspond at the correctdepth. Accordingly, the presence of occlusions can introduce a strongsource of error into a depth map.

Processes for generating depth maps in accordance with many embodimentsof the invention attempt to minimize sources of error that can beintroduced into a depth map by sources including (but not limited to)those outlined above. A general process for generating a depth map inaccordance with an embodiment of the invention is illustrated in FIG. 4.The process 400 involves capturing (402) a light field using an arraycamera. In a number of embodiments, a reference viewpoint is selected(404). In many embodiments, the reference viewpoint is predetermined. Inseveral embodiments, the reference viewpoint can be determined basedupon the captured light field or a specific operation requested by auser of the array camera (e.g. generation of a stereoscopic 3D imagepair). Prior to determining a depth map, the raw image data isnormalized (406) to increase the similarity of corresponding pixels inthe captured images. In many embodiments, normalization involvesutilizing calibration information to correct for variations in theimages captured by the cameras including (but not limited to)photometric variations and scene-independent geometric distortionsintroduced by each camera's lens stack. In several embodiments, thenormalization of the raw image data also involves pre-filtering toreduce the effects of aliasing and noise on the similarity ofcorresponding pixels in the images, and/or rectification of the imagedata to simplify the geometry of the parallax search. The filter can bea Gaussian filter or an edge-preserving filter, a fixed-coefficientfilter (box) and/or any other appropriate filter. In a number ofembodiments, normalization also includes resampling the captured imagesto increase the similarity of corresponding pixels in the capturedimages by correcting for geometric lens distortion, for example.Processes performed during the normalization or raw image data inaccordance with embodiments of the invention are discussed furtherbelow.

An initial depth map is determined (408) for the pixels of an imagecaptured from the reference viewpoint. The initial depth map is used todetermine (410) likely occlusion zones and the depths of pixels in theocclusion zones are updated (412) by determining the depths of thepixels in occlusion zones using images in which a corresponding pixel isvisible. As is discussed further below, depth estimates can be updatedusing competing subsets of images corresponding to different visibilitypatterns encountered in real world scenes. Although a specific sequenceis shown in FIG. 4, in many embodiments occlusion zones are detected atthe same time the initial depth map is generated.

A normalization process involving resampling the raw image data toreduce scene-independent geometric differences can reduce errors bycorrecting linear and/or non-linear lens distortion which mightotherwise compromise the ability to match corresponding pixels in eachof the captured images. In addition, updating the depth map in occlusionzones with depth measurements that exclude occluded pixels furtherreduces sources of error in the resulting depth map. Although a generalprocess for generating a depth map is illustrated in FIG. 4, variationsand alternatives to the illustrated processes for generating depth mapscan be utilized in accordance with embodiments of the invention.Processes for calibrating raw image data, determining initial depthmaps, and for updating depth maps to account for occlusions inaccordance with embodiments of the invention are discussed furtherbelow.

Increasing Similarity of Corresponding Pixels in Captured Image Data

The greater the similarity between the images captured by each of thecameras in an array camera, the higher the likelihood that a measurementof corresponding pixels in the images at different hypothesized depthswill result in highest similarity being detected at the correct depth.As is disclosed in U.S. patent application Ser. No. 12/935,504(incorporated by reference above) the images captured by cameras in anarray camera typically differ in a number of ways including (but notlimited to) variations in the optics from one camera to another canintroduce photometric differences, aliasing, noise, andscene-independent geometric distortions. Photometric differences andscene-independent geometric distortions can be corrected throughfiltering and calibration. Photometric calibration data used to performphotometric normalization and scene-independent geometric correctionsthat compensate for scene-independent geometric distortions can begenerated using an off line calibration process and/or a subsequentrecalibration process. The photometric calibration data can be providedto a photometric normalization module or process that can perform any ofa variety of photometric adjustments to the images captured by an arraycamera including (but not limited to) pre-filtering to reduce theeffects of aliasing and noise, Black Level calculation and adjustments,vignetting correction, and lateral color correction. In severalembodiments, the photometric normalization module also performstemperature normalization. The scene-independent geometric correctionsdetermined using a calibration process can also be applied to thecaptured images to increase the correspondence between the images. Whenthe captured images are used to synthesize a higher resolution imageusing super-resolution processing, the scene-independent geometriccorrections applied to the images are typically determined at asub-pixel resolution. Accordingly, the scene-independent geometriccorrections are typically determined with a higher degree of precisionthan the corrections utilized during registration in conventionalstereoscopic 3D imaging. In many embodiments, the scene-independentgeometric corrections also involve rectification to account fordistortion and rotation of the lenses of the array camera relative tothe focal planes so that the epipolar lines of the non-reference imagesare easily aligned with those of the image captured from the referenceviewpoint. By normalizing geometrically in this way, the searchesperformed to determine the depths of corresponding pixels can besimplified to be searches along straight lines in various cameras, andthe precision of depth measurements can be improved.

Systems and methods for calibrating array cameras to generate a set ofscene-independent geometric corrections and photometric corrections thatcan be applied to images captured by an array camera in accordance withembodiments of the invention are described in U.S. Patent ApplicationSer. No. 61/780,748, entitled “Systems and Methods for Calibration of anArray Camera” to Mullis, Jr., filed Mar. 13, 2013, the disclosure ofwhich is incorporated by reference in its entirety.

In a number of embodiments, the correspondence of the pixels in thecaptured images is increased by resampling the images to detect objectsto sub-pixel precision shifts in the fields of view of the cameras inthe array camera.

A process for applying corrections to images captured by an array camerato increase the correspondence between the captured images in accordancewith embodiments of the invention is illustrated in FIG. 5. The process500 includes photometrically normalizing the captured images (502),applying scene-independent geometric corrections (504) to the normalizedimages. In some embodiments, an additional rectification process (505)is needed to ensure that all cameras are co-planar and parallax searchcan be reduced to epipolar lines only. The processes shown in FIG. 5increase the correspondence between the resulting images. Therefore,searches for pixel correspondence between the images are more likely toresult in accurate depth measurements.

Although specific processes for increasing the correspondence betweenimages captured by an array camera(s) in accordance with embodiments ofthe invention are discussed above with respect to FIG. 5, any of avariety of processes that increase the correspondence between thecaptured images can be utilized prior to generating a depth map inaccordance with embodiments of the invention. Processes for generatingdepth maps in accordance with embodiments of the invention are discussedfurther below.

Generating a Depth Map

The process of generating a depth map involves utilizing disparitybetween images to estimate the depth of objects within a scene. As notedabove, occlusions can impact the reliability of depth measurementsobtained using cost functions in the manner outlined above. Typicallysuch occlusions will manifest themselves as significant mismatchesaccording to the similarity metric used to compare corresponding pixels(potentially masking the similarity of the visible pixels). However,many embodiments of the invention generate an initial depth map and thenaddress any errors that may have been introduced into the creation ofthe initial depth map by occlusions. In several embodiments, the initialdepth map is utilized to identify pixels in the image captured from areference viewpoint that may be occluded in images captured by the arraycamera from other viewpoints. When an occlusion is detected, the depthinformation for the pixel in the image captured from the referenceviewpoint can be updated by excluding pixels from the image in which thepixel is occluded from the similarity comparisons. In severalembodiments, depth estimates impacted by occlusions can be updated usingcompeting subsets of images corresponding to different visibilitypatterns encountered in real world scenes. In certain embodiments, theupdated depth estimates can be utilized to identify corresponding pixelsthat are occluded and the depth estimation process iterated using thevisibility information so that the impact of occlusions on the precisionof the depth map can be reduced. In several embodiments, the process ofgenerating updated depth estimates using subsets of images issufficiently robust that the need to iteratively refine the depth mapand visibility estimates can be reduced or eliminated.

A process for determining depth of pixels in an image captured from areference viewpoint in accordance with an embodiment of the invention isillustrated in FIG. 6. The process 600 includes determining (602) aninitial depth map using some or all of the images captured by an arraycamera. The visibility of each pixel in the image captured from thereference viewpoint in each of the captured images is then determined(604). Where a corresponding pixel location is occluded, the depth ofthe pixel in the image captured from the reference viewpoint can berecalculated (606) excluding the image in which the corresponding pixellocation is occluded from the cost function. A decision (608) is madeconcerning whether to continue to iterate. As depth measurements inocclusion zones are refined, additional information is obtainedconcerning the visibility of pixels within the occlusion zones in eachof the captured images. Therefore, repeating the recalculation of thedepths of pixels in the occlusion zones as the visibility information isrefined can iteratively improve the precision of the depth map. Any of avariety of termination conditions appropriate to a specific applicationcan be utilized to determine when to terminate the iterative loopincluding (but not limited to) the completion of a predetermined numberof iterations and/or the number of pixels for which updated depthinformation is determined in a specific pass through the iterative loopfalling below a predetermined number. In several embodiments, a singleiteration only is performed due to the exploitation of subsets of theset of images corresponding to real world visibility patterns to updatedepth estimates generated using mismatched pixels.

Once a finalized depth map is obtained, the visibility of each of thepixels in the captured images is determined (610) and the depth mapand/or visibility information can be utilized for a variety of purposesincluding but not limited to the synthesis of a high resolution imageusing super-resolution processing.

The computational complexity of a process similar to the processillustrated in FIG. 7 depends on the number of images compared whenperforming depth determinations. The further a camera is from thereference viewpoint the larger the disparity that will be observed. Inaddition, the furthest cameras in the array encompass all the othercameras within their envelope. Typically larger magnitude shifts enabledepth to be determined with greater precision. Therefore, using a camerathat captures an image from a reference viewpoint and the cameras thatare furthest from that camera to determine depth information can improveprecision of the detected depth. In addition, using an aggregated costoriginating from cameras with various baselines and directions cansignificantly improve reliability of depth estimates due to increasedlikelihood of reducing periodicity in matches. In the case of a 5×5array (see FIG. 7), the central Green camera (700) can be utilized tocapture an image from a reference viewpoint and the image data capturedby the central camera can be compared to image data captured by theGreen cameras (702) located in the four corners of the array todetermine depth. In other arrays, images captured by any of a variety ofcombinations of cameras can be utilized to determine depth in accordancewith embodiments of the invention. As is discussed further below,selection of specific subsets of cameras can decrease the likelihoodthat a pixel in a reference image will be occluded in image datacaptured by other cameras in the subset.

Although a specific process for generating a depth map and/or visibilityinformation in accordance with an embodiment of the invention isillustrated in FIG. 6, any of a variety of processes can be utilizedthat involve determining an initial depth map and then refining thedepth map by detecting occluded pixels and updating the depthmeasurements to exclude occluded pixels. Specific processes fordetermining depth and visibility of pixels in accordance withembodiments of the invention are discussed further below.

Determining an Initial Depth Map

Processes for determining the distance from an array camera to an objectin a scene involve locating the depth at which corresponding pixels inimages captured by the array camera have the highest degree ofsimilarity. As discussed above, at a specific depth a pixel in an imagecaptured from a reference viewpoint will shift a known distance along anepipolar line between the reference viewpoint and each of the cameras inthe camera array. The pixel in the image captured from the referenceviewpoint and the “shifted” pixels in the other images (i.e. the pixelsin the images located in locations determined based upon the anticipatedshift for a specific distance) are the corresponding pixels. When ahypothesized depth is incorrect, the corresponding pixels may exhibitvery little similarity (although in some scenes incorrect depths havehigh degrees of similarity due to features such as periodic texture).When the hypothesized depth is correct, the corresponding pixels willideally exhibit the highest degree of similarity of any of thehypothesized depths. When a depth map is used in super-resolutionprocessing of a captured light field, a depth map can be determined withsufficient precision to enable detection of sub-pixel shifts. Insuper-resolution processing, it is the scene-dependent shifts that areutilized and not the depth directly. Therefore, the ability to detectdepth corresponding to sub-pixel shift precision can significantlyimprove the performance of the super-resolution processing. The mannerin which resampling of the pixels of the captured images can be utilizedto determine depth with sub-pixel shift precision is discussed furtherbelow.

In many embodiments, a parallax search of a number of depths within arange in physical distance (e.g. 20 cm to infinity) is utilized toinform the disparities searched when performing depth estimation. Thesearch range can be divided into a number of depth indices such that theparallax shifts between consecutive depth indices is constant in pixelsfor a particular image and is set based upon a minimum sub-pixelprecisions as measured for the images captured by cameras in the arraycorresponding to the largest baselines with respect to the referenceviewpoint (see for example FIG. 7). This increases the likelihood ofsufficient accuracy in the depth estimates for use as inputs to asuper-resolution process. In other embodiments, consecutive depthindices need not correspond to constant pixel shifts and the depthsearch can adapt based upon the characteristics of the scene.

In several embodiments, a cost function is utilized to determine thesimilarity of corresponding pixels. The specific cost function that isused typically depends upon the configuration of the array camera, thenumber of images captured by the array camera, and the number of colorchannels utilized by the array camera. In a number of embodiments, thearray camera includes a single color channel and/or a depth map isgenerated using cameras within a single color channel. Where image datafrom within a single color channel is utilized to generate the depthmap, a cost function can be utilized that measures the variance of thecorresponding pixels. In several embodiments, sums of L1 norms, L2norms, or some other metrics can be used. For example, the aggregationof similarity metrics with respect to a target (typically reference butnon-reference may also be used). The smaller the variance, the greaterthe similarity between the pixels.

Image data from multiple spectral channels can also be utilized togenerate a depth map. In several embodiments, the depth at a given pixellocation is estimated by looking at the similarity of correspondingpixels from images within each of the spectral channels. In a number ofembodiments, the process of determining the depth at a given pixellocation also involves using information concerning the similarity ofcorresponding pixels from images across different spectral channels.Cost functions that can be utilized when generating a depth map usingimage data captured using multiple color channels include (but are notlimited to) L1 norms, L2 norms, or a combination of L1 and L2 norms, ofthe combinations of image data from the different color channels and/orthe variance/standard deviation of corresponding pixels within multipleindividual color channels. In other embodiments, truncated versions ofthe L1 and L2 norms and/or any block-based similarity measure based onrank, census, correlation, and/or any other appropriate metric such asthose practiced in multiview stereo disparity detection techniques canbe utilized.

As is discussed further below, many embodiments of the invention utilizesubsets of cameras including cameras from multiple color channelsgrouped based upon characteristics of natural scenes when determiningthe depth of a pixel location in an image form a reference viewpoint todecrease the likelihood that a given pixel location is occluded in thealternate view images captured by the other cameras in the subset ofcameras. Where an array camera utilizes a Bayer filter in the camerathat captures an image from the reference viewpoint, then a variety ofcost functions can be utilized to determine pixel similarity including(but not limited to) cost functions that measure the combination of Redvariance, Green variance, and Blue variance. In addition, different costfunctions can be applied to the pixels in different regions of an image.In several embodiments, a depth map is generated from image datacaptured by a central Green camera and a cluster of Red, Blue and Greencameras in each of the four corners of a camera array using thistechnique (see for example FIG. 7).

A process for determining the depth of a pixel using images captured byan array camera in accordance with an embodiment of the invention isillustrated in FIG. 8. The process 800 includes selecting (802) aninitial hypothesized depth or distance d for a selected pixel from animage captured from a reference viewpoint. Based upon the location ofthe pixel within the reference image and information concerning thebaseline between the reference viewpoint and the viewpoints of the othercameras used to perform the depth measurement, the corresponding pixellocations in each of the captured images at the hypothesized depth d aredetermined (804). In many embodiments, the input images to the parallaxdetection process are not geometrically corrected, and the geometriccorrection is applied on-the-fly by adding a vector offset to theparallax shift during the search to identify corresponding pixels at agiven depth d. In other embodiments, the geometric correction is appliedto the images before the search commences during a normalization processand no geometric correction vector must be added during the search whencalculating pixel correspondences (i.e. the geometric corrections arepre-calculated). In the latter case, the pre-correction of geometricdistortion can make the algorithm significantly more efficient onparallel processors such as SIMD and GPUs.

As noted above, occlusions can introduce errors into depth estimates.When occlusion/visibility information is available, occluded pixels canbe disregarded (806) as part of the depth measurement. When informationconcerning the visibility of pixels is not available (e.g. during thegeneration of an initial depth map and/or during the generation of adepth estimate using a subset of images), the similarity of all of thepixels in the corresponding pixel locations is used to determine depth.As is discussed below with reference to FIGS. 8A-8I, initial depthsearches can be performed with respect to image data captured fromsubsets of images captured by the array camera to identify a specificsubset of cameras in which a given pixel in the reference image isvisible.

When the corresponding pixels have been identified, the similarity ofthe corresponding pixels can be measured (808). In many embodiments, thesimilarity of the pixels is determined using a cost function. Thespecific cost function utilized depends upon the pixel information thatis compared. As noted above, in one embodiment, when pixels from asingle color channel are compared the cost function can consider L1norms, L2 norms, and/or the variance of corresponding pixels. Whenpixels from multiple color channels are compared, more complex costfunctions can be utilized including (but not limited to) cost functionsthat incorporate the L1 and/or L2 norms of the image data from multiplecolor channels and/or the variance/standard deviation of correspondingpixels within multiple individual color channels. In other embodiments,truncated versions of the L1 and L2 norms and/or any block-basedsimilarity measure based on rank, census, correlation, and/or any otherappropriate metric such as those practiced in multiview stereo disparitydetection techniques can be utilized. In several embodiments, theprocess of determining similarity utilizing a cost function involvesspatially filtering the calculated costs using a filter such as (but notlimited to) a fixed-coefficient filter (such as a Gaussian filter), orin an alternative embodiment, an edge-preserving filter. In the latterembodiment, filtering with an edge-preserving filter in this way is aform of adaptive support that utilizes information from photometricallysimilar neighboring pixels to improve the depth estimates. Without thefiltering the depth measurements are pixel-wise and are noisier than ifthey are filtered. Smoothing the cost function using adaptive supportcan prevent the generation of incorrect depths. In a number ofembodiments, the calculated costs are spatially filtered using abilateral filter, where the bilateral filter weights are determined fromthe reference image but, in contrast to a normal bilateral filter, theresulting filter weights applied to the calculated costs and not thereference image. In this way the reference image data can be used as aguide to improve the denoising of the cost estimates. In a number ofembodiments, a box filter and/or any other filter appropriate to therequirements of a specific application can be utilized.

The calculation of the cost function for corresponding pixels atdifferent depths is repeated sweeping across a range of hypothesizeddepths (812) until the depth search is complete (810). The most likelydepth can then be determined (814) as the hypothesized depth at whichthe (filtered) cost function indicates that the corresponding pixelshave the highest level of similarity. In several embodiments, for agiven depth computation early termination can occur if a single camerashows a very high mismatch. In this condition, the process can skip ontothe next hypothesized depth since match at the current depth would beunacceptable. In many embodiments, the process of performing depthsampling (i.e. comparing pixels in alternate view images based upon thedisparity at a specific depth) involves sampling depth uniformly indisparity space. Stated another way, depth samples can be taken atuniform pixel shifts along an epipolar line. In a number of embodiments,the search does not involve uniform sampling in disparity space. Inseveral embodiments, the search exploits image characteristics toincrease the efficiency of the search. In several embodiments, thesearch uses prior information about where objects are in the scene, suchas from a coarser or lower spatial resolution depth map or reducedsearch resolution in disparity (e.g. from an image preview), todetermine or restrict which depths are sampled in trying to form ahigher resolution depth map. For example, a preview depth map may beused to determine that there are no objects beyond a particulardistance, in which case for the depth search, no depth samples would beallocated beyond that distance.

Many images exhibit regions of similar color, therefore, the search forthe most likely hypothesized depth can be performed intelligently byselecting a first set of hypothesized depths that are more coarselydistributed across the range of possible hypothesized depths and thenlocating the depth among these that exhibits the highest degree ofsimilarity. A second search can then be performed to refine within amore granular range of depths around the depth that exhibited thehighest degree of similarity in the first set of depths. In the eventthat the more granular search fails and the best pixel found is not froma region exhibiting similar color, a full search can be performed acrossthe entire range of depths at more precise intervals than in theoriginal first coarse search. However, if a satisfactory match is foundin the second search, the depth that exhibits the highest level ofsimilarity within the second search can be used as the most likelyhypothesized depth.

In many embodiments, searches for the most likely depth of a pixel areperformed utilizing depth information determined for adjacent pixels. Inseveral embodiments, the search is performed by searching around thedepth of one or more adjacent pixels, by searching around a depthdetermined based on the depths of adjacent pixels (e.g. based on theaverage depth of adjacent pixels or based on linear interpolations ofpairs adjacent pixels) and/or by searching around a previouslyidentified depth (e.g. a depth determined with respect to a previewimage and/or a previous frame in a video sequence). Searching in thisway can also simplify the application of spatial filters whendetermining depth (see discussion below). In other embodiments, any of avariety of techniques can be utilized to reduce the computationalcomplexity of locating the most likely depth of the pixels in an image.

Although specific processes for determining the depth of a pixel in animage captured from a reference viewpoint are discussed above withrespect to FIG. 8, any of a variety of processes can be utilized todetermine the depth of a pixel including process that determine thedepth of a pixel from a virtual viewpoint based upon a plurality ofimages captured by an array camera. Processes similar to the processillustrated in FIG. 8 can be utilized to generate an initial depth mapand then to refine the depth map by ignoring images in which acorresponding pixel to a pixel location in an image from the referenceviewpoint is occluded. Processes for determining pixel correspondenceusing adaptive support in accordance with embodiments of the inventionare discussed further below.

Determining Pixel Correspondence in the Presence of Occlusions

Wherever there is a depth transition or discontinuity in the referenceviewpoint, pixels adjacent the depth transition are likely to beoccluded in at least one of the images captured by the array camera.Specifically, the pixels adjacent to the transition that are further indistance from the camera are likely to be occluded by the pixelsadjacent the camera that are closer to the camera. Ideally, a depth mapis determined using an aggregated cost function CV(x,y,d) for eachvisible camera i in the array that excludes occluded pixels as follows:

${{CV}( {x,y,d} )} = {\sum\limits_{i}\frac{{{Cost}^{i,{Ref}}( {x,y,d} )} \times {V^{i,{Ref}}( {x,y} )}}{{number}\mspace{14mu} {of}\mspace{14mu} {visible}\mspace{14mu} {cameras}\mspace{14mu} {at}\mspace{14mu} ( {x,y} )}}$

where Cost^(i,Ref)(x,y,d) is a similarity measure (i.e. the costfunction),

-   -   d is depth of pixel (x,y), and    -   V^(i,Ref)(x,y) is the visibility of pixel (x,y) and initially        V^(i,Ref)(x,y)=1 for all cameras.

In a number of embodiments, the individual costs Cost^(i,Ref)(x,y,d) arecomputed based on each disparity hypothesis d for each pixel (x,y) forcameras i, Ref as follows:

Cost^(i,Ref)(x,y,d)=S{I ^(i)(x,y,d),I ^(Ref)(x,y,d)}

where S is the similarity measure (for example), and

-   -   I^(i) is the calibrated image i after geometric calibration.

In several embodiments, the process of generating an aggregated cost caninvolve use of images to which the scene-dependent geometric shiftscorresponding to a specific hypothesized or candidate depth are appliedto all pixels in the image. In this way, a shifted image can begenerated for each candidate depth searched. Using the shifted images,an aggregated cost at each depth for a specific pixel location (x,y) inan image from the reference viewpoint can be generated in the manneroutlined above utilizing the similarity between the shifted images andthe reference image. In addition, the aggregated cost can consider thesimilarity of the shifted images at the candidate depth as follows:

${{CV}( {x,y,d} )} = {\Sigma_{k \in K}\frac{( {x,y} ){{Cost}^{k,{Ref}}( {x,y,d} )} \times V^{k,{Ref}}}{{number}\mspace{14mu} {of}\mspace{14mu} {cameras}\mspace{14mu} {in}\mspace{14mu} K} \times \Sigma_{i,{j \in L}}\frac{{{Cost}^{i,j}( {x,y,d} )} \times {V^{i,{Ref}}( {x,y} )} \times {V^{j,{Ref}}( {x,y} )}}{{number}\mspace{14mu} {of}\mspace{14mu} {pairs}\mspace{14mu} {of}\mspace{14mu} {cameras}\mspace{14mu} {in}\mspace{14mu} L}}$

Where K is a set of cameras in the same spectral channel as thereference camera,

-   -   L is a set of pairs of cameras, where both cameras in each pair        are in the same spectral channel (which can be a different        spectral channel to the reference camera where the light field        includes image data in multiple spectral channels),

Cost^(k,Ref)(x,y,d)=S{ImageRef(x,y),ShiftedImage^(k)(x,y,d)}, and

Cost^(i,j)(x,y,d)=S{ShiftedImage^(i)(x,y,d),ShiftedImage^(j)(x,y,d)}

In a number of embodiments, the sets K and L do not necessarily containall cameras or pairs of cameras that satisfy the requirements in K andL. Furthermore, the cumulative cost function can also be constructedusing a cost term in which the set of L includes arbitrarily largegroups of cameras for which the cost of corresponding pixels isdetermined. In many embodiments, the similarity metric S is the L1 norm.In several embodiments, the similarity metric can be any of a number ofwell known similarity metrics including (but not limited to) the L2norm, the variance or standard deviation of the corresponding pixels(particularly where L includes larger groups of cameras) window-basedsimilarity metrics incorporating correlation, rank, census and/or anyother measure appropriate to the requirements of a specific application.Although comparisons are discussed above in the context of shiftedimages, as can be readily appreciated comparisons can be performed byapplying shifts to individual pixel locations and comparingcorresponding pixels at a hypothesized depth (as opposed to applyingshifts to all pixels in an image and then comparing the shifted images).

In a number of embodiments, the cost function can also considersimilarity between corresponding pixels across different spectralchannels. In several embodiments, the similarity of neighborhoods ofpixels in pixels from different spectral channels can be evaluated usingany of a variety of metrics including (but not limited to) thecross-correlation of the pixels in the neighborhoods, the normalizedcross-correlation between the pixels in the neighborhoods and/or anyother metric for measuring the similarity of the relative values of twosets of pixels such as (but not limited to) entropy measures includingmeasuring mutual information.

In several embodiments, different weightings can be applied to thesimilarity of corresponding pixels within a spectral channel containinga reference image and the reference image, the similarity ofcorresponding pixels within alternate view images in the same spectralchannel, and/or the similarity of corresponding pixels within images indifferent spectral channels.

As discussed above, the aggregated cost function can be spatiallyfiltered as follows:

FilteredCV(x,y,d)=Filter_(x) _(n) _(,y) _(n) _(,εN(x,y)){Cost(x _(n) ,y_(n) ,d)}

where the Filter is applied in a neighborhood N(x,y) surrounding pixellocation (x,y).

The filter can be a simple 3×3 or N×N box filter or some other filterincluding (but not limited to) a joint bilateral filter that uses thereference image as guidance, a fixed coefficient filter (such as aGaussian filter, or a box filter), or any appropriate edge preservingfilter. In several embodiments, the weighted aggregated cost function isas follows:

${{FilteredCV}( {x,y,d} )} = {\frac{1}{Norm}{\sum\limits_{{({x_{1},y_{1}})} \in {N{({x,y})}}}{{{CV}( {x_{1},y_{1},d} )} \times {{wd}( {x,y,x_{1},y_{1}} )} \times {{wr}( {{I_{Ref}( {x,y} )} - {I_{Ref}( {x_{1},y_{1}} )}} )}}}}$

where N(x,y) is the immediate neighborhood of the pixel (x,y), which canbe square, circular, rectangular, or any other shape appropriate to therequirements of a specific application,

-   -   Norm is a normalization term,    -   I_(Ref)(x,y) is the image data from the reference camera,    -   wd is a weighting function based on pixel distance, and    -   wr is a weighting function based on intensity difference.

In many embodiments, the filter is a bilateral filter and wd and wr areboth Gaussian weighting functions.

Based upon the filtered aggregated cost function, a depth map can becomputed by selecting the depth that minimizes the filtered cost at eachpixel location in the depth map as follows:

D(x,y)=argmin_(d){FilteredCV(x,y,d)}

When the aggregated cost function is filtered using an edge preservingfilter in the manner outlined above, the likelihood that noise willresult in the incorrect detection of occluded pixels is reduced. Insteadof computing depths for individual pixels, an adaptive support window isused around each pixel to filter noise in a manner that preserves depthtransitions. Utilizing a filter such as (but not limited to) a bilateralfilter provides an adaptive window of support that adapts based upon thecontent. In many embodiments, a bilateral filter is used in which thereference image is used to define the spatial and range support for thebilateral filter (i.e. the parameters that define the size of the windowof pixels that contribute to the aggregated cost function for a specificpixel). As a result, smoothing of the cost function of a pixel can beachieved using the calculated cost function of pixels that are part ofthe same surface. In other embodiments, filters such as (but not limitedto) box filters are less computationally complex and provide sufficientfiltering for the requirements of specific applications.

Determining Pixel Correspondence for Pixels in Multiple SpectralChannels

Array cameras in accordance with many embodiments of the inventioninclude cameras in multiple spectral channels such as, but not limitedto, Red, Green and Blue cameras. The cost metric CV(x,y,d) is describedabove in the context of a single spectral channel and multiple spectralchannels. In the case of an array camera including Red, Green, and Bluecameras, the cost function can consider the similarity of pixels in theGreen cameras, the similarity in pixels in the Red cameras, and thesimilarity of pixels in the Blue cameras at a particular depth. Where acamera in a specific color channel is chosen as the reference camera(e.g. a Green camera), pixels in the other channels (e.g. Red and Bluecameras) are difficult to directly compare to pixels in the referenceimage. However, the disparity at a particular depth can be determinedand the intensity values of corresponding pixels in other color channelscan be compared. Incorporating these additional comparisons into thedepth estimate can improve depth estimates by utilizing informationacross all color channels. Various cost functions that can be utilizedto perform depth estimation in array cameras that include Red, Green,and Blue cameras are discussed further below. As can be readilyappreciated, however, the same cost functions can be utilized withrespect to any set of spectral channels in accordance with embodimentsof the invention.

In several embodiments, image data is captured using an array cameraincluding Red, Green and Blue cameras and a Green camera is selected asa reference camera. A cost function can be utilized that considers pixelcorrespondence between pixels in a set of Green cameras, between pixelsin a set of Red cameras, and between pixels in a set of Blue cameraswhen determining depth estimates. In several embodiments, the followingcost function can be utilized:

Cost(x,y,d)=γ_(G)(x,y)·Cost_(G)(x,y,d)+γ_(R)(x,y)·Cost_(R)(x,y,d)+γ_(B)(x,y)·Cost_(B)(x,y,d)

where Cost_(G)(x,y,d) is the measure the similarity of pixels inlocations within a set of Green cameras determined based upon the depthd and the location of the pixel (x,y) in the reference Green camera,

-   -   Cost_(R)(x,y,d) is the measure of the similarity of        corresponding pixels in locations within a set of Red cameras        determined based upon the depth d and the location of the pixel        (x,y) in the reference Green camera,    -   Cost_(R)(x,y,d) is the measure of the similarity of        corresponding pixels in locations within a set of Blue cameras        determined based upon the depth d and the location of the pixel        (x,y) in the reference Green camera, and    -   γ_(G), γ_(R), and γ_(B) are weighting factors for the Green, Red        and Blue cost functions respectively which may be constants for        the entire reference viewpoint, or may vary spatially.

The spatial weighting may depend on the captured image data (for exampleusing edge gradients), may correct or use known properties of the sensor(for example using a noise model prior for a given sensor to calculateSNR), as well as properties of the cost function (which is another casewhere the spatial weighting depends on the image data). Additionally,imaging parameters utilized during the capture of image data can also beconsidered in determining the weightings, such as (but not limited to)the gain or detected light level at which the image is captured, can beused to modulate the weighting factors.

The cost function Cost_(G)(x,y,d) can be one of the metrics describedabove. In many embodiments, Cost_(G)(x,y,d) uses a similarity measurebased upon an L1 norm comparing a pixel in an alternate view image witha pixel in the reference image, an L2 norm comparing a pixel in analternate view image with a pixel in the reference image, and/orvariance across the pixels in the set of images captured by the Greencameras. In other embodiments, truncated versions of the L1 and L2 normsand/or any block-based similarity measure based on rank, census,correlation, and/or any other appropriate metric such as those practicedin multiview stereo disparity detection techniques can be utilized.

In a number of embodiments, the cost functions for the other colorchannels (i.e. Cost_(R)(x,y,d) and Cost_(B)(x,y,d)) do not utilize acomparison that includes a pixel from the reference image as the basisof determining pixel correspondence. In several embodiments, thesimilarity of corresponding pixels are performed by calculating theaggregated difference between each unique pair of corresponding pixelsin the set of cameras within the color channel. In the example of anarray camera in which depth is determined using four Red cameras, R_(A),R_(B), R_(C), and R_(D), the cost can be determined as follows:

Cost_(R)(x,y,d)=|R _(A)(x _(A) ,y _(A))−R _(B)(x _(B) ,y _(B))|+|R_(A)(x _(A) ,y _(A))−R _(C)(x _(C) ,y _(C))|+|R _(A)(x _(A) ,y _(A))−R_(D)(x _(D) ,y _(D))|+|R _(B)(x _(B) ,y _(B))−R _(C)(x _(C) ,y _(C))|+|R_(B)(x _(B) ,y _(B))−R _(D)(x _(D) ,y _(D))|+|R _(C)(x _(C) ,y _(C))−R_(D)(x _(D) ,y _(D))|

where (x_(A),y_(A)), (x_(B),y_(B)), (x_(C),y_(C)), and (x_(D),y_(D)) arepixel locations determined based upon the disparity in each of thecameras R_(A), R_(B), R_(C), and R_(D) respectively at depth d.

The above metric can be referred to as the combination cost metric andcan be applied within any color channel that does not contain thereference camera. In several embodiments, a combination metric can beutilized that does not include all combinations of unique pairs ofcorresponding pixels in the set of cameras within the color channel. Inseveral embodiments, unique pairs of corresponding pixels from a subsetof the images captured by an array camera can be utilized. When depthsare determined for a virtual viewpoint, none of the spectral channelscontain the “reference camera” and the combination cost metric can beapplied in each of the spectral channels. Although the combination costmetric is shown above utilizing the L1 norm to determine the similaritybetween pixel intensity values, in other embodiments, the L2 norm, thepixel variance, truncated versions of the L1 and L2 norms and/or anyblock-based similarity measure based on rank, census, correlation,and/or any other appropriate metric such as those practiced in multiviewstereo disparity detection techniques can be utilized.

Weighting factors (e.g. γ_(G), γ_(R), and γ_(B)) can be used todetermine the contribution of each of the spectral channels to a depthestimate. The weights can be fixed or vary from pixel to pixel (i.e.spatially-varying) in the reference image. In many embodiments, a map ofsignal-to-noise ratio (SNR) can be generated with respect to thereference image using an SNR estimator. In several embodiments, the SNRestimator can determine SNR based upon a prior characterization ofsignal noise. Areas where the SNR response is high can indicate thepresence of texture or high signal. Areas where the SNR estimate is lowcan indicate a textureless region, consisting almost entirely of noise.In certain situations, the data from the images might be noisy incertain spectral channels, but not in others. For example, an area mayappear textureless in Green images, but have signal content in imagescaptured by Red cameras. Therefore, the Green cameras will contributelittle useful information to the cost function and may actuallyintroduce noise into the depth estimation process, resulting in a lessreliable depth estimate than if only Red or Blue cameras were includedin the cost function. Therefore, an SNR map can be utilized to determineweightings to apply to each of the color channels. In severalembodiments, if the SNR estimate in the reference image for a pixel(x,y) is low, meaning that the immediate region around pixel (x,y) islikely textureless and does not contain significant signal, then theweighting for the color channel containing the reference image should bereduced at the pixel (x,y).

In many embodiments, a stricter condition can also be used and/or usedas an alternative in which the weighting for the spectral channelcontaining the reference image should be reduced at the pixel (x,y),when the SNR estimate in the reference image at a pixel (x,y) and forthe radius of maximum parallax (along epipolar lines) in the referenceimage for all of the cameras show low SNR, then the weighting for thespectral channel containing the reference image should be reduced at thepixel (x,y). The radius of maximum parallax can be determined only withrespect to pixels located along epipolar lines determined with respectto the other cameras in the camera array within the spectral channel.The stricter criterion acknowledges that though the SNR may be low atthe pixel location (x,y) in the reference image, there may be contentsome distance away (less than a maximum parallax shift) from pixel (x,y)in another camera within the color channel containing the referencecamera which could disqualify a candidate depth from being a likelymatch. Therefore, though the pixel location (x,y) may have low SNR,nearby content may still provide useful information to disqualifyingcertain depths as possibilities.

In a number of embodiments, strong SNR in the reference image may beused to reduce the weighting applied to the other color channels to savecomputation (i.e., fewer cameras must be searched). In addition, the SNRmay be estimated for a camera in one of the other color channels todetermine the weighting that should be applied to the color channel inthe cost function. In many embodiments, the process of determining theSNR involves estimating SNR along the epipolar line which connects thepixel location (x,y) in the alternate view camera to the referencecamera. Then, the epipolar line or line(s) may be searched for regionsof high SNR. If high SNR contributions are found in the alternate viewcamera along the epipolar line, the weighting of the color channel towhich the alternate view camera belongs can be set so that the colorchannel contributes to the cost metric for the pixel location (x,y) inthe reference image. If, instead, along the epipolar line beginning atthe pixel location (x,y), the alternate view image shows only low SNR,then the contribution of the color channel containing the alternate viewimage can be correspondingly reduced. In many embodiments, multiplecameras in each color channel are considered when determining theweighting to apply to the color channel when determining depthestimates. Although specific processes for estimating depth usinginformation contained within color channels that do not include thereference camera are described above, any of a variety of processes canbe utilized to determine depth estimates based upon informationcontained in multiple color channels as appropriate to the requirementsof specific applications in accordance with embodiments of theinvention.

Based upon the initial depth map, visibility V^(i,Ref)(x,y) can beupdated based upon the computed depth map D(x,y) or based upon afiltered depth map F(D(x,y)) that is filtered using either a fixedcoefficient filter (such as a Gaussian or a box filter), or adaptive oredge preserving filter such as (but not limited to) a joint bilateralfilter that uses the reference image as guidance. A variety oftechniques can be utilized for determining whether a pixel in an imagecaptured from a reference viewpoint is occluded in another image. In anumber of embodiments, an initial depth map is utilized to identifypixels that may be occluded. Information concerning foreground objectscan be utilized to identify zones in which occlusions are likely tooccur. In addition, any of a variety of additional characteristics of aninitial depth map can be utilized to detect occlusions including (butnot limited to) unusual depth transitions within the depth map and/orpixel depth values that are inconsistent with local pixel depth values.

Although much of the discussion above with respect to FIG. 8 relates togeneration of depth estimates using image data in a single spectralchannel or in the Red, Green, and Blue color channels, the techniquesdescribed above are equally appropriate with respect to any of a varietyof spectral channels (the terms color channel and spectral channelsbeing used herein interchangeably). An aggregated cost function similarto those described above can be utilized including a cost termdetermined with respect to each spectral channel using any of thetechniques described above. In addition, the cost function can includecost terms that weight the similarity of pixels across spectral channelsusing techniques similar to the those described above. The accuracy ofresulting depth estimates can depend upon the extent to which the depthestimate utilizes pixels from an image in which a pixel location (x,y)in a reference image is occluded. Techniques for improving the accuracyof depth estimates when a pixel location (x,y) in an image from areference viewpoint are occluded within one or more images in the set ofimages are discussed further below.

Generating a Depth Map Accounting for Occlusions Using Subsets of Images

Patterns of visibility occur in natural scenes. Therefore, the patternV^(i,Ref)(x,y) is typically not arbitrary. A strong prior existsconcerning the cameras in which a pixel in the reference image isvisible. In many embodiments, a depth map can be determined in a mannerthat also provides an estimate for V^(i,Ref)(x,y) in which there is alow likelihood that cameras in which pixel (x,y) is occluded areincorrectly identified. Based upon the estimate for V^(i,Ref)(x,y), adepth estimate can be determined without the need to iterate to refinethe visibility of V^(i,Ref)(x,y). Alternatively, additional iterationscan be performed to refine the depth map by including additional camerasbased upon visibility information obtained using a reliable depthestimate. By obtaining a better initial depth map, however, theiterative process is likely to converge more rapidly.

In many embodiments, the process of generating a depth map involvesdetermining depth using multiple clusters or subsets of cameras thateach correspond to a different pattern of visibility within the sceneand selecting the depth estimate as the depth determined using thesubset of images captured by the cluster of cameras in whichcorresponding pixels have the highest similarity. A process fordetermining a depth for a pixel (x,y) using images captured by groups ofcameras representing subsets of a camera array in accordance with anembodiment of the invention is illustrated in FIG. 8A. The process 850includes selecting (852) an initial group of cameras corresponding to aspecific pattern of visibility within the scene and determining (854) acandidate depth estimate using the subset of image data captured by thegroup of cameras is generated based upon the depth at whichcorresponding pixels within the group of cameras have the highestsimilarity. In several embodiments, the process is similar to thatoutlined above with respect to FIG. 8 with the exception that the costsare determined for each subset of images and the lowest cost depthestimate generated using the subsets is selected as a candidate depthestimate for a relevant pixel location. As is discussed further below,the similarity of corresponding pixels within the subset of images atthe candidate depth estimate is then compared against the similarity ofcorresponding pixels within other subsets of images at other candidatedepth estimates to determine the candidate depth estimate that is mostreliable within the context of a given application. In many embodiments,the candidate depth estimate that is selected as the depth estimate forthe pixel location is determined based upon the subset of images havingthe most similar corresponding pixels at the candidate depth estimate.

In many embodiments, the group of cameras can include cameras frommultiple color channels and a cost metric weighting similarity in eachcolor channel is utilized to estimate the depth of pixel (x,y) in thereference image using the group of cameras. The process then iterates(856, 858, 854) through a plurality of different groups of cameras thateach correspond to different patterns of visibility within the sceneuntil a depth estimate is determined for each of the groups of cameras.The depth estimate for pixel location (x,y) in an image from thereference viewpoint can be obtained by selecting the depth estimate fromthe group of cameras in which the corresponding pixels at the estimateddepth have the highest similarity. As noted above, similarity of pixelscan be determined using a cost function that weights similarity ofpixels in multiple spectral channels and/or across spectral channels.The subset of images in which the corresponding pixels at the estimateddepth have the highest similarity corresponds to a specific pattern ofvisibility and provides an initial estimate of V^(i,Ref)(x,y) that has alow likelihood of incorrectly identifying that the pixel location (x,y)in an image from the reference viewpoint is visible in a camera in whichit is occluded.

Although the discussion provided above is presented in the context ofperforming searches with respect to each group of cameras with respectto pixel locations (x,y) in the reference image, depth maps can beseparately estimated for the pixels in the reference image using eachgroup of cameras corresponding to a specific visibility pattern. In thisway, the cost metrics determined for pixel location (x,y) using aparticular group of cameras can be filtered to smooth out noise in thecost function. Therefore, the depth estimate for the pixel location(x,y) can be selected using the depth estimate for the group of camerashaving the smallest filtered cost metric. The filters applied to thecost metrics determined using a specific group of cameras can be fixed,or can be spatially adaptive. The specific filters utilized can bedetermined based upon the requirements of specific applications inaccordance with embodiments of the invention. Following selection of thedepth estimates for the pixels in the reference image, additionalfiltering can be performed to further smooth noise in the initial depthmap.

The clusters or groupings of cameras utilized to detect particularpatterns of visibility within a scene can depend upon the numbers ofcameras in an array camera, the camera that is selected as the referencecamera, and/or the distribution of cameras from different color channelswithin the array. Eight groups of cameras in a 5×5 array correspondingto different patterns of visibility that are likely to be present withina scene with respect to pixels in a reference camera located at thecenter of the array are shown in FIGS. 8B-8I. The eight groups aregenerated by rotating and flipping the same group template, whichincludes 12 cameras. Depending upon the orientation of the grouptemplate, this includes seven Green cameras, and either three Redcameras and 2 Blue cameras, or 3 Blue cameras and 2 Red cameras. Asnoted above, the group template can be utilized to select groups ofcameras when estimating depth for a pixel (x,y) in a reference Greencamera located at the center of the 5×5 array. The depth of the pixellocation (x,y) can be estimated by selecting the depth estimate from thegroup of cameras in which the corresponding pixels in the three colorchannels at the estimated depth have the highest similarity.

Although specific groups are shown in FIGS. 8B-8I for selecting groupsof cameras, any of a variety of templates corresponding to commonvisibility patterns within a scene can be utilized. For example, groupsof cameras along a single epipolar line can be selected as describedbelow with reference to FIGS. 10. In many embodiments the groups areselected so that the same number of cameras in the color channelcontaining the reference camera appears in each group of cameras. Inaddition, the groups can be determined so that there are at least twocameras in the other color channels in each group of cameras. If thegroups include an uneven number of cameras, then the cost metric withrespect to different sized groups may be biased and the bias can beaccounted for through normalization. In many embodiments, the groups ofcameras are selected to provide baseline diversity (contrast with thegroups illustrated in FIG. 10 that are selected based upon sharing acommon baseline). The greater the number of different radial epipolarlines on which depth searches are performed, the more likely one of theimages captured by a group of cameras will contain information that canassist in identifying incorrect depths. In several embodiments, thegroup of cameras are selected so that the central angle of the sectordefined by the epipolar lines of each group is the same.

In smaller array cameras, such as (but not limited to) 4×4 arraycameras, and depending upon the pattern of color filters utilized withinthe array it may not be possible to select groups of cameras thatcontain the same number of cameras in each color channel. In severalembodiments, a color filter pattern is utilized so that groups ofcameras corresponding to common visibility patterns contain the samenumber of cameras in a single color channel. In this way, image datacaptured within the color channel can be utilized to estimate depths foroccluded or otherwise mismatched pixels by comparing the filtered costsof depth estimates obtained using the different subgroups. Four groupsof cameras in a 4×4 array corresponding to different patterns ofvisibility that are likely to be present within a scene with respect topixels in a reference camera located at the center of the array areshown in FIGS. 8J-8M. The four groups are generated by rotating andflipping the same group template, which includes 4 cameras. In theillustrated embodiment, there are three color channels: Red, Green, andBlue. Each group of cameras includes three Green cameras and one Blue orRed camera. Due to the presence of a single Red or Blue camera, inseveral embodiments depth estimates are determined using the image datacaptured in the Green color channel. For a given pixel location (x,y),the image data in the Red or Blue camera of the group that yielded themost reliable depth estimate (i.e. the lowest cost) is assumed visiblein an image from the reference viewpoint. Accordingly, the pixel valuein the pixel location in the Red or Blue image corresponding to thepixel location (x,y) in the image from the reference viewpoint can beutilized as a reference pixel for the purpose of calculating thevisibility of corresponding pixels in other images within the Red orBlue color channels. For each of the groups shown in FIGS. 8J-8M, one ofthe spectral channels is excluded from the group. The use of a π filtergroup can, however, be utilized to identify which of the images in theexcluded color channel should be used as a reference image for thepurpose of determining the visibility of pixels in the excluded colorchannel. When the viewpoint of a central camera of a π camera group isutilized as a reference viewpoint, two images in the excluded colorchannel will have been captured from viewpoints on opposite sides of thereference viewpoint. In typical natural scenes, a pixel location withinan image from the reference viewpoint is likely to be visible in atleast one of images captured by the adjacent cameras in the excludedcolor channel. In order to determine which (if any) of the images ismost likely to contain a corresponding pixel to a pixel location in animage from the reference viewpoint that is visible, the similarity ofthe corresponding pixels within the two subgroups that contain the twoimages. Assuming that the corresponding pixels in at least one of thesubgroups achieves a threshold level of similarity, then the image inthe subgroup in which the corresponding pixels have the highest level ofsimilarity can be selected as a reference image for the excluded colorchannel. In this way, the visibility of corresponding pixels in anyimage within the excluded color channel can be determined based upon itssimilarity with the corresponding pixel from the reference image for theexcluded color channel. Where neither image captured from the adjacentviewpoints to the reference viewpoint reliably contain a visible pixelcorresponding to a pixel location within an image from the referenceviewpoint, then alternative techniques can be utilized to identify animage within the excluded color channel that contains a correspondingpixel that is visible and/or to determine the visibility of pixelswithin individual images from the excluded color channel. In severalembodiments, visibility can be determined by performing epipolar linesearches in the manner described herein. In a number of embodiments,cross-channel similarity measures can be used to determine acorresponding pixel within the images in an excluded color channel thatcan be utilized as a reference pixel. In several embodiments, the imagein which the neighborhood surrounding the corresponding pixel exhibitsthe highest cross-correlation (or any other appropriate cross-channelsimilarity measure) with the reference image can be utilized as areference pixel for the purpose of determining the visibility of theother corresponding pixels in the images in the excluded color channel.A similar approach can be taken with array cameras including differentsized camera arrays.

In many embodiments, the groups of cameras used to estimate depth for apixel in a reference camera correspond to pairs of cameras within thearray. Through use of thresholds, cameras in which a pixel (x,y) islikely to be visible can be identified and an initial visibility mapV^(i,Ref)(x,y) constructed. The threshold can be a hard threshold,and/or a threshold based upon the SNR in the reference image. The sameis also true of larger groups of cameras such as those illustrated inFIGS. 8B-8I. Thresholds can be used to detect the presence of one ormore outlier pixels within a set of corresponding pixels. Groups ofcameras that are found to not contain outliers can then be combined, andthe depth recalculated with this new combined set, to improve theprecision of depth estimates. In a similar manner, an initial depth mapcan be constructed by initially assuming that all cameras are visible inthe visibility map V^(i,Ref)(x,y). Any of a variety of techniques can beutilized to determine whether a pixel (x,y) is occluded in at least oneof the cameras in the camera array including (but not limited to) use ofthresholds in the manner outlined above, and/or performing occlusionsearches along epipolar lines. The depth of pixels that are likely to beoccluded in at least one of the cameras in the array can then beestimated again using a process similar to the process outlined abovewith respect to FIG. 8A. In this way, the cameras in which the pixel isoccluded can be rapidly identified.

Although a variety of processes for determining depth maps andvisibility maps when estimating depth for pixels within a referenceimage are described above with reference to FIGS. 8A-8I, any of avariety of processes can be utilized to determine an initial depth mapand/or visibility map in accordance with the requirements of specificapplications in accordance with embodiments of the invention. Processesfor identifying occluded pixels including processes that involveperforming searches for occluded pixels along epipolar lines inaccordance with embodiments of the invention are discussed furtherbelow.

Identifying Occluded Pixels

A challenge associated with identifying occluded pixels from an initialdepth map is that depths within the depth map that are determined usingoccluded pixels may be incorrect. The most reliable depth estimates arethose of the objects in the scene that are closest to the camera. Theseare the objects that also give rise to the greatest disparity and canpotentially result in the largest number of pixel occlusions (dependingupon the distribution of objects within the scene. Therefore, adetermination can be made concerning whether a pixel visible in thereference image is occluded in a second image by searching for anoccluding pixel in the reference image along the baseline vector. Anoccluding pixel is a pixel that is sufficiently close to the camera thatthe disparity observed from the perspective of the second image would besufficiently large as to occlude the pixel visible in the referenceimage. The search for occluding pixels can be understood with referenceto FIG. 9. An image captured from a reference viewpoint 900 is shown inFIG. 9. In order to determine the visibility of pixel (x₁,y₁) with adepth d₁ in a second image captured by the array camera, a search isconducted along a line 902 parallel to the baseline between the camerathat captured the reference image and the camera that captured thesecond image. The pixel (x₁,y₁) will be occluded, when a pixel (x₂,y₂)is closer to the camera (i.e. located at a depth d₂<d₁). Therefore, allpixels (x₂,y₂) where d₂≧d₁ can be disregarded. Where the scene-dependentgeometric shifts of each pixel (s₂ and s₁ respectively) are greater thanthe distance between the two pixels along the line 902 parallel to thebaseline vector, then pixel (x₂,y₂) will also occlude the pixel (x₁,y₁).Stated another way, pixel (x₂,y₂) occludes pixel (x₁,y₁) when

|s ₂ −s ₁−√{square root over ((x ₂ −x ₁)²+(y ₂ −y ₁)²)}|≦threshold

In several embodiments, the threshold in the above expression can bedetermined as the inverse of the super-resolution factor used duringsubsequent super-resolution processing (e.g. when the super-resolutionprocess attempts to achieve an increase in resolution of a factor of 3,then a threshold of ⅓ of a pixel is appropriate). When no pixel can befound satisfying the above expression, then the pixel (x₁,y₁) can beconcluded to be visible in the second image. Where the above expressionis satisfied for some pixel (x₂,y₂), then the pixel (x₁,y₁) can beconsidered to be occluded. Therefore, the process illustrated in FIG. 8can be repeated to create an updated depth estimate for the pixel (x,y)disregarding the second image (and any other images in which the pixelis occluded). As can readily be appreciated, incorrect depths in theinitial depth estimate can result in visible pixels being disregarded infuture iterations of the process for determining a depth map. Usingadaptive support to provide depths that are photometrically consistentdecreases the likelihood that noise will result in the detection offalse pixel occlusions, which eliminate useful information fromsubsequent process iterations. In many embodiments, the decision todesignate a pixel as being occluded considers the similarity of thepixels and the confidence of the estimated depths of the pixels (x₁,y₁)and (x₂,y₂). As is discussed further below, a confidence map can begenerated with respect to the depth map of the reference image and theconfidence map indicates the reliability of a specific depth map.Therefore, a possible occlusion identified using the expression providedabove in which scene-dependent geometric shifts of each pixel (s₂ and s₁respectively) are based upon unreliable depth estimates can bedisregarded. Similarly, a possible occlusion involving pixels where thedifference in the intensities of the pixels is below a predeterminedthreshold can be disregarded. Where the pixels values are sufficientlysimilar, a depth estimate generated in reliance on the pixel willlargely be unaffected even if the pixel is occluded. In otherembodiments, a variety of other considerations can be taken into accountwhen determining whether to indicate a pixel as occluded in a visibilitymap as appropriate to the requirements of specific applications.

The search discussed above with respect to FIG. 9 can be conducted alongthe epipolar line corresponding to every camera utilized to perform thedepth estimation. When the captured images are rectified correctly, thesearch can be simplified by aligning the baselines of the camerasrelative to the rows and columns of the pixels captured from thereference viewpoint (see discussion of rectification above). The searchfor occluding pixels need not be performed with respect to every pixelin the reference image. Instead, an initial search can be conducted forpixels that are likely to be occluded in one or more images captured bythe array camera including (but not limited to) pixels proximate depthtransitions. Searches can then be performed for occluding pixels withrespect to pixels that are considered likely to be occluded. Inaddition, the search for occluding pixels can be performed moreefficiently by computing the projections of pixels based upon distancein advance (the projections indicate the depth at which adjacent pixelsalong the baseline will be occluded). In addition, once a pixel isdetermined to be occluded the process for detecting occlusion ofadjacent pixels can be simplified by utilizing the projection of theoccluding pixel. In other embodiments, any of a variety of techniquescan be utilized to more efficiently locate occluding and occluded pixelsincluding (but not limited to) rendering the image based on depth inaccordance with embodiments of the invention.

As noted above, including occluded pixels in the determination of theinitial depth map can introduce errors in the resulting pixel depths.When occlusions are detected using a process similar to any of theprocesses outlined above and the depth map updated, errors in the depthmap are removed. As errors in the depth map are removed, more accuratepredictions can be made as to the pixels that are occluded. Accordingly,the process of detecting occlusions can be performed iteratively until astopping criterion is reached. In a number of embodiments, the stoppingcriterion can be (but is not limited to) the number of occluded pixelsdetected in a specific iteration (that were not previously detected)falling below a predetermined number and/or the completion of apredetermined number of iterations.

Referring back to FIG. 6, a process for generating and refining a depthmap in accordance with an embodiment of the invention is illustrated. Inmany instances, the processes for determining (602) the initial depthmap will have a tendency to overestimate the disparity of occludedpixels. This has the effect of pushing occluded pixels into theforeground. Therefore, in a number of embodiments, a pixel that occludesanother pixel can also be treated like an occluded pixel for thepurposes of updating (606) the depth map. In this way, the depth ofbackground pixels that have incorrect initial depth measurements can bedetected and updated. As is discussed further below, the visibility ofpixels that are ignored can be separately determined (610) once thedepth map is finalized. In a number of embodiments, processes such asthe process 600 illustrated in FIG. 6 also involve application of abilateral filter to the depth map to help stabilize the depth map as theprocess iterates.

The accuracy of a depth estimate typically increases with the number ofimages in the light field captured by the array camera utilized ingenerating the depth estimate. Considering a smaller number of imagescan, however, reduce the computational complexity of obtaining a depthestimate. When a depth transition occurs, occlusions will typicallyoccur in images captured on one side of the reference viewpoint.Therefore, a search for occluding pixels similar to the search describedabove can be utilized to determine whether a pixel is occluded in agroup of images captured to one side of the reference camera. If thesearch indicates that no occlusions occurred, then the depth of thepixel can be determined using that group of images and the depth mapupdated.

A 5×5 array camera that can be utilized to construct a depth map usingthe Green cameras in the array is illustrated in FIG. 10. The arraycamera 1010 includes a central reference Green camera (1012). Theremaining Green cameras in the array can be utilized to form eightradial groups of three Green cameras for the purpose of determiningdepth of pixels that are occluded in at least one of the images capturedby the Green cameras. Although radial groups are illustrated in FIG. 10,groups of cameras in each of four quadrants surrounding the referenceviewpoint can also be utilized. A group may be as small as a pair ofcameras, one of which is the camera that captures an image from thereference viewpoint. In many embodiments, groups such as those discussedabove with reference to FIGS. 8A-8I can also be utilized.

Although specific processes for detecting pixel occlusions are discussedabove with respect to FIGS. 6, 8A-8I, 9, and 10, any of a variety ofprocesses can be utilized to generate a depth map including (but notlimited to) processes that reduce the computational complexity ofdetecting occlusions in accordance with embodiments of the invention. Inseveral embodiments, the process of determining the depth of each pixelcan involve searching based upon both hypothesized depths andhypothesized visibility and the combination of depth and visibility thatyields the highest pixel correspondence selected as the most likelydepth and set of occlusions. Visibility determined in this way can beconfirmed by using the approach described above for detecting occludingpixels.

In many embodiments, information concerning the visibility of pixels inthe captured images from the reference viewpoint is utilized inprocesses including (but not limited to) super-resolution processing.Processes for determining the visibility of pixels in images captured byan array camera from a reference viewpoint using a depth map inaccordance with embodiments of the invention are discussed furtherbelow.

Determining Visibility of Pixels

Pixel visibility can be utilized in determining a depth map and in avariety of other applications including (but not limited to)super-resolution processing. In several embodiments, a depth map for animage captured from a reference viewpoint generated using a processsimilar to the processes outlined above is utilized to generate avisibility map for the other images captured by an array camera (i.e.the images captured from alternate viewpoints). In several embodiments,visibility maps can be determined with respect to whether pixels inalternate view images are visible from the reference viewpoint andwhether a pixel in a first alternate view image is visible any of theother alternate view images. In a number of embodiments, the process ofdetermining visibility maps for the images captured within a singlecolor channel involves comparing the photometric similarity of pixelscorresponding to a pixel in the image captured from the referenceviewpoint. Pixels that are considered to have a predetermined level ofsimilarity can be considered visible and pixels that are below athreshold level of similarity can be considered occluded. The thresholdutilized to determine the photometric similarity of corresponding pixelscan adapt based upon the similarity of the corresponding pixels. Inseveral embodiments, the threshold is determined as a function of thephotometric distance between the pixel from the reference image and thecorresponding pixel that is most similar to the pixel from the referenceimage. When an array captures image data in multiple color channels, thevisibility of pixels in a single color channel can be utilized todetermine the visibility of pixels in other channels.

A process for determining the visibility of corresponding pixels to apixel within a reference image in accordance with an embodiment of theinvention is illustrated in FIG. 11. The process 1100 includes selecting(1102) a pixel from an image captured from the reference viewpoint. Adepth map generated using a process similar to the processes describedabove can be utilized to determine the depth of the selected pixel.Based upon the depth of the selected pixel, the locations of thecorresponding pixels in each image captured by the array camera can bedetermined (1104). The similarity of the corresponding pixels to theselected pixel from the reference image can be utilized to determine thevisibility of the corresponding pixels. In a number of embodiments, thephotometric distance of the pixels is utilized as a measure ofsimilarity and a threshold used to determine pixels that are likelyvisible and pixels that are likely occluded. In many embodiments, thethreshold varies depending upon the characteristics of the pixels thatare compared. In certain embodiments, the threshold value used todetermine similarity of corresponding pixels is determined using theintensity of a reference pixel, as the average of a subset of pixelintensity values for corresponding pixels that are found to be visiblein a given color channel. In several embodiments, the specificcorresponding pixel intensities that are averaged can depend uponcorresponding camera baseline and confidence values for the pixels (ifavailable) and associated matching costs for the pixels. In severalembodiments, the threshold is determined (1106) as a function of thephotometric distance between the selected pixel from the reference imageand the corresponding pixel that is photometrically closest to theselected pixel. In a number of embodiments, the threshold is based uponthe pixel intensity of the corresponding pixel in the reference imageand/or the intensity of the pixel in the alternate view image. Incertain embodiments, the threshold is determined using an SNR model forthe captured image. In a number of embodiments, the photometric distanceof the selected pixel and the closest corresponding pixel is scaledand/or an offset is added to obtain an appropriate threshold. In otherembodiments, any of a variety of techniques can be utilized fordetermining a threshold for determining the visibility of acorresponding pixel including (but not limited to) using a fixedthreshold.

The selected pixel from the reference image and the corresponding pixelsare compared (1108) and the threshold used to determine (1110) thesimilarity of the pixels. When the photometric distance of the selectedpixel from the reference image and one of the corresponding pixels isless than the threshold, then the corresponding pixel is determined(1112) to be visible. When the photometric distance of the selectedpixel from the reference image and one of the corresponding pixelsexceeds the threshold, then the corresponding pixel is determined (1114)to be occluded.

The process (1100) illustrated in FIG. 11 can be repeated for a subsetor all of the pixels in the reference image to generate visibility mapsfor the corresponding pixels in other images captured by the arraycamera. In embodiments where all of the pixels in the camera thatcaptures an image from the reference viewpoint are in a single colorchannel, then processes similar to the process illustrated in FIG. 11effectively generate visibility for images captured within a singlecolor channel. When the array camera also includes images captured inother color channels, the visibility of pixels in images that are not inthe same color channel as the reference image can be determined byperforming similar comparisons to those described above with respect tocorresponding pixels from images within the spectral channel that knownor are likely visible in the reference viewpoint. In other embodiments,the camera that captures the reference image employs a Bayer filter (oranother appropriate filter pattern) that enables the capture of imagedata in multiple color channels from the reference viewpoint. In whichcase, processes similar to those illustrated in FIG. 11 can be utilizedto generate visibility information for images in multiple colorchannels, where the process involves demosaicing the Bayer filterpattern to obtain a Red and Blue pixel value at each position in thereference view.

Although specific processes are discussed above in the context of FIG.11, any of a variety of processes can be utilized to determine thevisibility of pixels in images captured by an array camera in accordancewith embodiments of the invention including (but not limited to)processes that iteratively refine visibility information as part of theprocess of generating a depth map. In many embodiments, the process ofgenerating a depth map and a visibility map also includes generating aconfidence map that can provide information concerning the reliabilityof the estimated depths within the confidence map. Processes fordetermining confidence maps in accordance with embodiments of theinvention are discussed further below.

Confidence Maps

Processes for generating depth maps, including those described above,can result in regions within a depth map in which depth estimates areunreliable. A textureless region of an image synthesized using imagedata captured by an array camera is illustrated in FIG. 18A and thedepth map for the image generated using processes similar to thosedescribed above in accordance with embodiments of the invention isillustrated in FIG. 18B. In the textureless region 1800, the cost metricused to determine depth in the manner described above is noisy andthough a minimum cost (i.e., at a depth where the cameras show maximumcorrespondence) can be found, the result depends largely on sensor andphoton noise and not any significant underlying signal. The pixelcorrespondence in such regions (as measured by the cost metric) isindeed greatest at the depth shown, but the resulting depth shown is notthe correct depth of the object. In contrast, in the edge region 1802,the cost function shows a depth at which the cost is minimized withgreat certainty. There, the edge signal is much greater than the noiseand so the disparity corresponding to the actual depth of the object canbe detected with higher confidence.

Depth errors are not limited to textureless regions, however. Anotherclass of depth errors occurs in zones of occlusion, where certainbackground regions are visible in some cameras, and not others. Thissort of error can be seen along the rim of the tire, where theforeground region intersects the background region. In the depth map,there appear to be regions 1804 containing incorrect depth estimates atthe interface between the foreground to background.

When generating a depth map, a confidence map can be generated, whichdescribes numerically, through some measure, the reliability ofdifferent depth estimates within the depth map. The confidence map canbe used by later processing stages, or by third-party applications, todetermine which regions of the depth map can be most relied upon forfurther processing. For example, a depth measurement utility can allow auser to click on regions of an image synthesized using asuper-resolution process to obtain the depth of a particular pixel. Ifthe user clicks on a pixel of the image, the confidence map can bechecked before returning a result. If the confidence of the depth at therequested location is low, then the user interface can avoid reportingthe depth at that location. If the confidence map is high, then the userinterface can safely report the depth at the selected location. That is,the confidence map can be used to qualify the results for particularapplications and not return an inaccurate value where the depth map isknown to contain errors.

The confidence map can be encoded in a variety of ways, and there may bemultiple classes or axes of confidence encoded within a singleconfidence map. A variety of confidence measures that can be utilized toencode a confidence metric in a confidence map in accordance withembodiments of the invention are discussed below.

In several embodiments, a confidence map is encoded with a confidencemeasure based on whether the depth estimate of a pixel is within atextureless region within an image. As noted above, textureless regionscan be detected based upon SNR in the region surrounding a given pixel.If the SNR is above a known tunable threshold, the region may be markedtextureless in a binary fashion. Alternatively, the SNR itself (withoutbeing thresholded) may be remapped linearly or non-linearly and used toserve as a continuous indicator of confidence.

In many embodiments, a gradient edge map (e.g. Prewitt or Canny) may becalculated and used as a confidence metric within a confidence map.Since edges and texture typically have high confidence, gradient mapsand SNR maps often provide a good coarse measure of confidence for adepth map.

In a number of embodiments, the confidence map can be encoded based uponwhether a particular region is low confidence due to occlusions and/ormismatch and conflicting measurements between cameras (i.e. there may betexture in a region that is detected by an SNR map, but there may stillbe a depth error occurring because in that area the parallax detectionprocess detects and/or is unable to resolve occlusions or otherwiseconfidently estimate depth for any other reason).

In a number of embodiments, a confidence metric is determined as the“best cost” achieved during the depth estimation process, or a linear,non-linear, or quantized remapping of this quantity. If the minimum costachieved during depth estimation is above a selected threshold, theregion may be marked low confidence on the basis of a lack ofcorrespondence between multiple views at the estimated depth.

In a number of embodiments, occlusions may be detected by comparing thebest costs between different subgroups or depth maps generated betweendifferent groups of cameras and if the difference between best costs isgreater than a threshold, marking low confidence for the pixel locationswhere the costs found in subsets of images differ.

In a number of embodiments, the confidence map can be encoded based uponwhether a particular region is low confidence due to adaptive processingsteps in the depth estimation process itself. For example, if fewerdepths were searched in a particular region, this information can beencoded numerically in the confidence map to highlight that the depth isless reliable. In many embodiments, certain regions of the depth map areactually searched through correspondence search, and other regions ofthe depth map, the depths are interpolated based on results from a depthsearch on a sparser set of points. In such a case, the pixels withinterpolated depths are given lower confidence values than pixels wherethe correspondences have actually been searched.

In several embodiments, the expected precision of a depth measurementcan also be coded in the confidence map as a numerical quantity. In manyinstances, depths farther away from the camera are measured with greatererror and so should be less trusted. In such cases the confidence mapcan mark such areas as involving lower confidence depth estimates. Theconfidence can be proportional to the expected depth error betweenadjacent search positions at that depth. In certain embodiments, thedisparity corresponding to the minimum distance supported by theparallax search (i.e. this will be the maximum disparity observedbetween any two cameras for all supported depths) can be determined.Once the maximum disparity is found, the search will search a number ofdisparities up to the maximum disparity. In many embodiments the maximumdisparity is D low resolution pixels and the number of depths searchedis N. The number of pixels of disparity between adjacent searchpositions along an epipolar line is DI(N−1). The depth in meters for anyone of the N disparities that is searched (indexed by n<N) ared_(n)=CI(n*DI(N−1)) where C is a constant that incorporates informationabout the baselines and focal lengths of the individual low resolutioncameras having the maximum baselines. If, at a particular point in thedepth map, the depth is d_(n), then the expected measurement error atd_(n) is e_(n)=½*max(|d_(n)−d_(n+1)|, |d_(n)−d_(n−1)|). Basically, theexpected measurement error is the error due to searching a fixed,discrete number of points along the epipolar line. The error valueitself may be mapped linearly or non-linearly to provide a confidencevalue with respect to a depth estimate at a particular pixel locationwithin the depth map. The higher the error, the less confident the depthestimate. In several embodiments, disparities searched are not spacedequally, but may be coarser in some regions than others. Accordingly,error can be calculated similarly between adjacent indices (whatever thedistribution of search positions along the epipolar line) so that theconfidence map reflects the calculated error in depth. In a number ofembodiments, the confidence map reflects the maximum error in estimateddisparity (not depth), the inverse of the quantity listed above. Thismay be more useful for applications such as image fusion, whereas thedepth error would be more useful for measurement applications that occurin real world coordinates (such as, but not limited to, 3D modeling).

In a number of embodiments, the confidence map can mark regions as lowconfidence due to known or detected lens or sensor defects that make thedepth map unreliable. Defective pixels (a term that includes bothdefective pixels on the sensor as well as pixels affected by lensdefects) may be detected during image processing or offline in apre-processing calibration step. In one embodiment, if the total numberof pixel defects within a radius of a particular pixel (x,y) in anyreference camera exceeds a pre-set threshold, the pixel (x,y) is markedlow confidence in the depth map due to pixel defects. In anotherembodiment, a similar confidence value may be defined where confidenceincreases proportionally (not as a hard threshold) as a function of thenumber of pixel defects in any camera within a radius and/or regionsurrounding the reference pixel (x,y) (or pixels known to be affected bylens defects). In another embodiment, the confidence may be apre-calculated value for specific configurations of defective pixelsthat are known to create errors of varying severity. In severalembodiments the confidence value for the defect takes into account thelocal image content in calculating the confidence value.

In several embodiments, the confidence map may mark as low confidenceareas where the reference image appears textureless but in other colorchannels there is textured content. In one embodiment, a pixel (x,y) inthe reference camera is searched within a local radius and/or region. Ifwithin this local radius and/or region the content is considered to betextureless in Green, but if the same search within another (perhapslarger) local radius/region for Red and/or Blue cameras turns upsufficient texture in images within the Red and/or Blue color channels,the region can be marked as lower confidence due to the fact that theGreen color channel is less useful in this detection scenario and depthresults will be less reliable (though often correct).

In a number of embodiments the confidence map numerically encodes as lowconfidence scenarios in which there is photometric mismatch due to lensflare or features in the scene. In many embodiments, the localstatistics (mean and variance) of a region of interest around the pixellocation (x,y) may be calculated and compared to the local statistics ofa similar region in another camera. In this way, local image statisticsbetween two neighborhoods in the same general region of multiple imagescan be compared to detect possible lens flare, the presence of whichreduces confidence. In other embodiments, any of a variety of techniquescan be utilized to compare neighborhoods in multiple images to detectlens flare. The resulting confidence measure can be a scaled ornon-linearly mapped function of the difference between the mean andvariance of the regions across images captured by multiple cameras. Thegreater the mean and variance differences between the images captured bythe cameras, the less likely the depth is reliable and the lower theconfidence will be.

In a number of embodiments the confidence map adapts to lightingconditions to reduce the confidence when the image is very dark andnoisy. In certain embodiments, the sensor gain at the time the image wastaken will result in an absolute reduction in confidence for all depths.In another embodiment, the analog gain and exposure time of the sensorare taken into account when computing a SNR ratio, or thresholds foredge gradients at different levels of noise. In many embodiments, theanalog gains and exposure times for different focal planes can beutilized for different cameras in a camera array used to capture a setof images.

To detect regions which are of low confidence due to occlusions, thebest-achieved cost metric may be stored during the parallax search. Forregions which show significant occlusion, the best achieved cost metricusually greatly exceeds the minimum value that would occur if there wereno occlusion and all cameras saw the same content. Accordingly, athreshold can be applied to the best achieved costs. If the bestachieved cost exceeds the threshold, then the region is marked as likelyto have been occluded or to have had some other problem (such asphotometric non-uniformity).

For certain similarity metrics, the low-confidence threshold forocclusion can be corrected for the mean intensity of the region as wellas the noise statistics of the sensor. In many embodiments, the mean ofthe region in the reference image is calculated using a spatial box N×Naveraging filter centered around the pixel of interest. In otherembodiments, once the mean is known, the noise statistics for the colorchannel containing the reference camera may be calculated by a tablelookup which relates a particular mean at a particular exposure and gainto a desired threshold. If the best matching value greatly exceeds theexpected noise, then the pixel can be marked as unreliable due topossible occlusion.

A non-binary measure of confidence due to general mismatch can beobtained using the following formula:

Confidence(x,y)=F(Cost_(min)(x,y),Cost^(d)(x,y),I(x,y)^(cam),Sensor,Cameraintrinsics)

where Cost_(min)(x,y) is the minimum cost of a disparity search over thedesired depth range,

Cost^(d)(x,y) denotes that cost data from any depth or depths (besidethe minimum depth),

I(x,y)^(cam) image data captured by any camera can be utilized toaugment the confidence;

Sensor is the sensor prior, which can include known properties of thesensor, such as (but not limited to) noise statistics orcharacterization, defective pixels, properties of the sensor affectingany captured images (such as gain or exposure),

Camera intrinsics is the camera intrinsic, which specifies elementsintrinsic to the camera and camera array that can impact confidenceincluding (but not limited to) the baseline separation between camerasin the array (affects precision of depth measurements), and thearrangement of the color filters (affects performance in the occlusionzones in certain scenarios).

In several embodiments, Confidence(x,y) may make use of valuesneighboring pixel location (x,y) (i.e. spatial neighborhoods) for allthe arguments.

In many embodiments, a confidence map can be encoded based upon factorsincluding (but not limited to) one or more of the above factors. Eachfactor may be encoded in a binary fashion or may be represented as arange of (quantized) degrees of confidence, or may be non-quantizedranges or derivatives thereof. For example, the confidence along aparticular axis may be represented not as a single bit, but as multiplebits which represent the level of confidence that a region istextureless. In certain embodiments the confidence along a particularaxis may be represented as a continuous or approximately continuous(i.e. floating point) quantity. Other factors considered whendetermining confidence can be determined using any of a variety ofranges of degrees of confidence as appropriate to the requirements ofspecific applications in accordance with embodiments of the invention.In several embodiments, an arbitrary number of confidence codes orvalues are included in a confidence map for a particular pixel where onemay specify any or all of these conditions. Specific confidence metricsare discussed further below.

In a particular embodiment where only the minimum cost is considered andnoise statistics of the sensor follow a linear model, a simplified formmay be used:

${{Confidence}( {x,y} )} = {{a \times \frac{{Cost}_{\min}( {x,y} )}{{Avg}( {x,y} )}} + {offset}}$

where Avg(x,y) is the mean intensity of the reference image in a spatialneighborhood surrounding (x,y), or an estimate of the mean intensity inthe neighborhood, that is used to adjust the confidence based upon theintensity of the reference image in the region of (x,y),

-   -   a and offset are empirically chosen scale and offset factors        used to adjust the confidence with prior information about the        gain and noise statistics of the sensor.

In this case, higher values would indicate lower confidence, and itwould be up to the image processing pipeline to determine how tothreshold the results.

In general, the confidence map provides metadata describing the depthestimates contained within the depth map that quantifies numerically theaccuracy of detected depths in the image. In many embodiments, theconfidence map may be provided in an external delivery format along withthe depth map for use with the depth map in applications including (butnot limited to) machine vision, gesture recognition, post capture imagerefocusing, real-time applications, image special effects,super-resolution, or other applications. An example of the manner inwhich a confidence map can be utilized in a depth estimation process tofilter a depth map in accordance with an embodiment of the invention isillustrated in FIGS. 18C-18H. Turning first to FIG. 18C is an image of ascene containing objects at different depths synthesized using asuper-resolution process from multiple images captured in differentcolor channels (specifically Red, Green and Blue color channels). Adepth map generated from the reference viewpoint using processes similarto those outlined above is illustrated in FIG. 18D. As can be readilyappreciated, the depth map is noisy. A confidence map generated usingany of a variety of the metrics outlined above can be generated as partof the process of generating a depth map. A binary confidence mapgenerated by thresholding SNR in accordance with an embodiment of theinvention is illustrated in FIG. 18E. An 8-bit confidence map generatedbased upon SNR in accordance with an embodiment of the invention isillustrated in FIG. 18F. A binary confidence map generated by combininga confidence factor generated by thresholding SNR and a confidencefactor generated by thresholding the number of corresponding pixels thatare occluded in accordance with an embodiment of the invention isillustrated in FIG. 18G. In several embodiments, the confidence map canbe utilized to filter the depth map. A depth map that is filtered basedupon a binary confidence map generated by thresholding SNR in accordancewith an embodiment of the invention is illustrated in FIG. 18H.Comparing the depth maps shown in FIGS. 18D and 18E reveals the value ofthe use of the confidence map in interpreting depth informationgenerated using any depth estimation process. Although a specificconfidence metric and filtering process are described above withreference to FIGS. 18C-18H, any of a variety of confidence metrics canbe utilized in the filtering and/or additional processing of depthestimates and/or depth maps in accordance with embodiments of theinvention. The generation of confidence maps and the use of confidencemaps to filter depth maps in accordance with embodiments of theinvention is further illustrated in the close up images shown in FIGS.18I-18L. With specific regard to FIG. 18I, a close up image of an objectsynthesized from a light field of images captured in Red, Green, andBlue color channels (each image containing image data in a single colorchannel) using super-resolution processing is shown. A depth mapgenerated using the techniques outlined above is illustrated in FIG.18J. A binary confidence map generated by thresholding SNR generated inaccordance with an embodiment of the invention is illustrated in FIG.18K. A multibit resolution confidence map generated in accordance withan embodiment of the invention using SNR is illustrated in FIG. 18L. Abinary confidence map generated by thresholding the SNR of the regionsurrounding each pixel and by thresholding the number of pixels in theimages within the light field that correspond to a pixel location in theimage from the reference viewpoint that are occluded in accordance withan embodiment of the invention is illustrated in FIG. 18M. A depth mapfiltered using the binary confidence map shown in FIG. 18M isillustrated in FIG. 18N.

In several embodiments, a confidence map generated using one or more ofthe metrics described above can be inserted as an additional channel ofinformation into the JPEG-DZ file format or other file formats. Inseveral embodiments, the confidence map is encoded and decoded usingprocesses similar to those outlined in U.S. patent application Ser. No.13/631,731 to Venkataraman et al. entitled “Systems and Methods forEncoding Light Field Image Files”, filed Sep. 28, 2012. The disclosureof U.S. patent application Ser. No. 13/631,731 is herein incorporated byreference in its entirety. Although specific metrics for determiningconfidence are described above, any of a variety of metrics fordetermining confidence appropriate to the requirements of a specificapplication can be utilized in accordance with embodiments of theinvention.

Reducing Computational Complexity

A variety of strategies can be utilized to reduce the computationalcomplexity of the processes outlined above for determining depth mapsand for determining the visibility of images captured by a camera array.In several embodiments, a depth map is constructed by only searching fordepth at a reduced (i.e. sparse) subset of pixel locations. The depthsearch is done at fewer points (i.e. a sparser set of points in theimage) and for points that depth is not calculated, the depth isassigned through other means. By the end, this sparse depth searchprovides a depth for every pixel location in a reference image wheresome pixels are searched and others are filled in through interpolation.As previously stated, not every pixel in the final depth map has a depthobtained by comparing the similarity of the pixel to correspondingpixels in the captured images. Instead, in regions where nocorrespondence search is done, the depths of many of the pixels aredetermined using processes including (but not limited to) averaging thedepths of surrounding pixels (where the correspondence search has beenrun) or interpolating the depths of adjacent pixels which have beencalculated. By reducing the number of pixels for which depthmeasurements are performed, the amount of computation used to generate adepth map can be reduced. In several embodiments, the amount ofcomputation used when detecting a depth map can also be reduced bydetecting textureless areas of the image and using processes including(but not limited to) assigning a single depth value from the nearestindicator pixel where depth has been calculated, averaging the depths ofsurrounding pixels or interpolating the depths of adjacent pixels todetermine the depth of pixels in the textureless areas of the image. Inother embodiments, any of a variety of processes for reducing thecomputational complexity of generating a depth map can be utilized asappropriate to the requirements of specific applications in accordancewith embodiments of the invention including varying the precision of thedepth estimates within the depth map based upon characteristics of thescene including (but not limited to) regions containing edges, and/orbased upon object distance. Processes for generating depth maps fromsparse depth searches and for detecting textureless regions in images inaccordance with embodiments of the invention are discussed furtherbelow.

Generating Depth Maps from Sparse Depth Search

Processes for generating depth maps through sparse search in accordancewith embodiments of the invention typically involve determining depth ofa sparse set of pixels spaced or distributed throughout the referenceimage. Based upon this initial depth map consisting of sparse points,depth transitions can be detected and the depths of pixels surroundingthe depth transitions can be directly measured using the processesoutlined above. The depths of the remaining pixels can be determinedbased upon the depths of sparse pixels in the depth map. In manyembodiments, the depth measurements are performed using a subset of thepixels in the reference image at the resolution at which they werecaptured.

A process for determining a depth map through sparse search inaccordance with an embodiment of the invention is illustrated in FIG.13. The process 1300 includes dividing (1302) the reference image intospatial blocks (or groups of associated pixels) and generating (1304)depth measurements for a sparser subset of indicator pixels within thespatial blocks. Here, spatial block may be taken to referinterchangeably to a rectangular block of pixels, or a subset ofassociated pixels that need not conform to any particular shape.

Indicator pixels are a subset of the pixels within the spatial block (orgroup of associated pixels) and are typically selected to provideinformation concerning variation in depth across the spatial block. Aspatial block 1400 including a plurality of indicator pixels 1402 inaccordance with an embodiment of the invention is illustrated in FIG.14. The indicator pixels 1402 are selected at the edges and at thecenter of the spatial block. Although specific indicator pixels areillustrated in FIG. 14, the arrangement of indicators within a spatialblock or group of associated pixels can be varied and/or any of avariety of pixels within a spatial block can be selected as indicatorpixels as appropriate to the requirements of a specific application. Ina number of embodiments, different shaped spatial blocks are utilizedand the shape of the spatial block can be varied. In severalembodiments, the arrangement of indicator pixels within the spatialblocks can be varied. In many embodiments, the indicator pixels areselected based upon scene content. In certain embodiments, the indicatorpixels are selected based on which points within the spatial block havethe highest SNR in the reference image to increase the likelihood thatthe points most likely to give confident depth results are used. Inanother embodiment, fixed spatial positions are chosen for someindicator pixels (as indicated in FIG. 14) for all blocks, and somesubset of indicator pixels are assigned to points with highest SNR inthe spatial block (i.e. a mixed configuration). In another embodiment, asegmentation process can be used to create relevant spatial regionsbased on scene content. Although a rectangular spatial block is shownother techniques could be used for splitting the image into spatialclusters, which contain some indicator pixels as described above.Furthermore spatial blocks can be larger in certain portions of theimage than in others.

Referring back to FIG. 13, depth can be assigned (1306) to the pixels ineach block based upon the depths of the indicator pixels. In severalembodiments, the assigned depth is obtained through interpolation of theneighboring indicator pixels. In several embodiments, the depth of anon-indicator pixel may be calculated as a normalized weighted averageof the distances to the nearest indicator pixels within a fixedneighborhood. Alternatively, nearest neighbor interpolation (1308) canbe utilized to assign depths to the pixels in the spatial block basedupon the depth measurements of the indicator pixels. In anotherembodiment, weights for the interpolation can incorporate intensitysimilarity as well as spatial distance to the nearest indicator pixels.In another embodiment, a non-linear regression such as (but not limitedto) a Kernel Regression may be used to fill in the missing positionsbetween depths sampled at the indicator pixel positions. In anotherembodiment, a single depth for the entire block is assigned byminimizing the summed costs of the indicator pixels within the block. Inother embodiments, any of a variety of techniques can be utilized togenerate depth information for pixels within a spatial block.

In many embodiments, the reliability of each of the spatial blocks inthe depth map is determined (1310). Within the spatial block, depthswill have been estimated both for indicator pixels (where search hasoccurred) and non-indicator pixels (where depths have been interpolatedbased on indicator pixel results). For the indicator and non-indicatorpixels, the costs of the estimated depths within the block aredetermined. The costs of each pixel in the block are summed to create areliability indicator. If the total cost of all pixels within the blockis greater than a threshold, then the spatial block is marked asunreliable due to the fact that the estimated depths for some pixelsappear to have poor correspondence. Where a spatial block has beendetermined to have low reliability of poor spatial correspondence, thenthe block is likely to contain a depth transition or occlusion. If suchis the case, then the full correspondence search and occlusionprocessing can be run within the spatial block.

If a spatial block is determined to have a depth transition per thecriteria above, then the spatial block may be ‘split’ and new setsindicator pixels selected in each of the two child spatial blocks andthe process iterated. In one embodiment, the block may be split in half.In another embodiment, the block may be split into unequal regionsdepending on the depths solved by the indicator pixels within thespatial block.

Where depth transitions are detected within and/or between spatialblocks, the depth map can be refined (1312) by performing additionaldepth measurements within the spatial block that contains the depthtransitions. In this way, the computational complexity of generating thedepth map is reduced by reducing the number of depth measurementsperformed in generating an accurate depth map.

Although a specific process for generating a depth map from sparsesearches in accordance with embodiments of the invention is illustratedin FIG. 13, any of a variety of processes that generate a depth map byperforming fewer depth measurements in regions of similar or slowlytransitioning depth can be utilized in accordance with embodiments ofthe invention.

Reducing Computation in Textureless Regions of Images

In many embodiments, the process of generating a depth map involvesreducing the amount of computation needed for textureless regions of theimage. Textureless areas can be ambiguous with parallax, because thecorresponding pixels at many hypothesized depths may be similar.Therefore, depth measurements within a textureless area can generateunreliable and noisy results. In many embodiments, the SNR in the regionsurrounding a pixel is used when determining the depth of the pixel toidentify whether the pixel is in a textureless area. An initial depthestimate or a set of initial depth estimates for a given pixel can bedetermined based upon the depth of at least one adjacent pixel for whicha depth has previously been determined. When the variance of thecorresponding pixels for the given pixel (or any other similaritymeasure) is below the SNR threshold in the region surrounding the pixel,the pixel can be assumed to be part of a textureless area and (one of)the approaches described below can be used to select the depth of pixel.Otherwise, a depth measurement can be performed using a process similarto the processes described above.

In many embodiments, textureless regions may be detected using a fixedthreshold on the SNR. The computation for the search in such regions maybe reduced by reducing the number of depths searched. In manyembodiments, the full set of depths will be searched until a minimumcost depth is identified that is below a noise-dependent threshold thattakes into account the noise characteristics of the sensor. When theminimum cost is found to be below the threshold the depth is accepted asthe depth of the textureless region and no more depths are searched(i.e. the search is terminated as soon as a depth that has “closeenough” correspondence is found). In many embodiments, the search intextureless regions may save computation by searching the full range ofdisparity but at larger increments than are done in the normal searchfor a region with texture (i.e. reducing the number of depthssearched)—the best cost will be selected as the depth of the pixel inthe textureless region.

A process for detecting textureless regions using the SNR surrounding apixel in accordance with an embodiment of the invention is illustratedin FIG. 15. The process 1500 includes selecting (1502) a pixel from thereference image and detecting (1504) the SNR in the region around theselected pixel. An initial hypothesized depth d can be determined (1506)for the pixel. In many embodiments, the initial hypothesized depth d isdetermined based upon the depth of one or more pixels in the regionsurrounding the selected pixel. A determination (1508) is then madeconcerning whether the variance or cost of the corresponding pixels atthe hypothesized depth is below a threshold that can be (but is notlimited to) predetermined or a function of the SNR in the regionsurrounding the selected pixel. In other embodiments, any of a varietyof similarity measures can be utilized to determine whether the regionsurrounding the pixel is textureless. In the event that variance or costof the corresponding pixels is below a noise or predetermined threshold,then the hypothesized depth is selected as the most likely depth on theassumption that the pixel is located within a textureless region. Whenthe variance or cost of the corresponding pixels exceeds the noise orpredetermined threshold, then the depth of a pixel is determined inaccordance with a process similar to the processes described above.

Although a specific process for detecting textureless areas within areference image are described above with respect to FIG. 15, any of avariety of processes for detecting textureless areas in an image can beutilized in accordance with embodiments. Furthermore, any of a varietyof processes can be utilized to detect other characteristics of an imagethat can be relied upon to reduce the number of depth measurements thatare made in generating a reliable depth map in accordance withembodiments of the invention.

Generating Depth Maps from Virtual Viewpoints

While much of the discussion provided above describes the generation ofdepth maps with respect to images captured by a reference camera,systems and methods in accordance with embodiments of the invention cansynthesize images from virtual viewpoints. A virtual viewpoint is areference viewpoint that does not correspond to the viewpoint of anycameras within a camera array. Accordingly, irrespective of the numberof color channels within a camera array, none of the color channelsinclude a camera in which image data is captured from the referenceviewpoint. An example of a virtual viewpoint that can be defined inaccordance with an embodiment of the invention is illustrated in FIG.12. The array camera module 1200 includes a 4×4 array of camerasincluding 8 Green cameras, 4 Red cameras, and 4 Blue cameras. A virtualcamera 1202 is defined with a virtual viewpoint at the center of thearray camera module. Although a specific virtual viewpoint isillustrated in FIG. 12, any virtual viewpoint can be arbitrarily definedwith respect to the cameras within a camera array.

When determining a virtual viewpoint depth map, there is no explicitreference camera which can be searched and used for cost metriccomparisons. In many embodiments, the depth of a given pixel (x,y) in animage synthesized from the virtual viewpoint is determined bycalculating the effective baseline from the virtual imager with respectto all other cameras in the array. The baseline for a camera at position(i,j) with respect to the virtual viewpoint would be (i,j)−(i_(v),j_(v))where (i_(v),j_(v)) is the location of the virtual viewpoint 1202. Oncethe baselines between the individual cameras is determined with respectto the virtual viewpoint, the process of estimating depth proceeds bysearching for depths at which corresponding pixels having the highestsimilarity. For each pixel (x,y) in the virtual reference camera (i.e.an image from the virtual viewpoint), the search can proceed much as inthe typical parallax scenario, where for each depth d, the disparitywith respect to each of the alternate view cameras is determined at thatdepth, and then the similarity of corresponding pixels in one or more ofthe color channels is determined using an appropriate cost metric. Inmany embodiments, the combination cost metric described above fordetermining the similarity of pixels in color channels that do notcontain the reference camera can be utilized. In many embodiments, acamera adjacent the virtual viewpoint in a specific color channel can beused as a reference camera for the purpose of determining the similarityof the pixel in the chosen reference camera with corresponding pixels inimage data captured by other cameras in the color channel. In manyembodiments, a Green camera is chosen as a reference camera for thepurpose of determining the similarity of corresponding Green pixels anda combination cost metric is used for corresponding pixels in othercolor channels. In many embodiments, the process of determining aninitial depth map for the virtual viewpoint can involve forming groupsof cameras corresponding to patterns of visibility within the scene in asimilar manner to that described above with respect to FIGS. 8A-8I.

A depth map generated for a virtual viewpoint can be utilized tosynthesize a high resolution image from a virtual viewpoint using asuper-resolution process in accordance with embodiments of theinvention. The primary difference in the synthesis of a high resolutionimage from a virtual viewpoint is that the high resolution grid is froma virtual viewpoint, and the pixels are fused to the high resolutiongrid using correspondences calculated with baselines which are withrespect to the virtual view position and not a physical referencecamera. In this case there is no physical reference camera having pixelsthat map regularly on the high resolution grid. As can be readilyappreciated, processes for determining confidence maps for depth mapswith respect to virtual viewpoints can be determined using similaraccommodations related to analyzing the synthesized reference image orchoosing an image close to the virtual viewpoint as a proxy forperforming SNR and/or other related measurements. Although specificprocesses for generating depth maps with respect to virtual viewpointsare described above, any of a variety of processes incorporating thecost metrics and techniques outlined above can be utilized to generatedepth estimates for virtual viewpoints in accordance with embodiments ofthe invention. Systems for performing parallax detection and correctionand for generating depth maps in accordance with embodiments of theinvention are discussed further below.

Systems for Performing Parallax Detection

A system for generating a depth map and visibility information usingprocesses similar to those described above is illustrated in FIG. 16.The system includes a parallax detection module 1600 that takes as aninput captured images that form a light field and calibrationinformation for an array camera and outputs a depth map, and theestimated visibility of the pixels of the captured images. In manyembodiments, the parallax detection module 1600 also outputs aconfidence map indicating the reliability of the depth measurements forspecific pixels within the reference image. As is discussed furtherbelow, the depth map, estimated visibility information, and/orconfidence map can be provided to a super-resolution processing modulewithin an array camera to generate a higher resolution image from thecaptured images and to any of a variety of applications that can utilizedepth, confidence and/or visibility information. In many embodiments,the parallax detection module and the super-resolution module areimplemented in software and/or firmware on a microprocessor within thearray camera. In several embodiments, the software associated with theparallax detection module and the super-resolution module is storedwithin memory within the array camera. In other embodiments, theparallax detection module and/or the super-resolution module can beimplemented using any appropriately configured hardware and/or software.The generation of high resolution images from a light field captured byan array camera using a depth map generated in accordance withembodiments of the invention is discussed further below.

Super-Resolution Processing Using Depth Maps

As is noted in U.S. patent application Ser. No. 12/967,807 (incorporatedby reference above) disparity between images can introduce significantartifacts when performing super-resolution processing. Therefore, thesuper-resolution processes disclosed in U.S. patent application Ser. No.12/967,807 can involve applying scene dependent geometric corrections tothe location of each of the pixels in the images captured by an arraycamera prior to using the images to synthesize a higher resolutionimage. The baseline and back focal length of the cameras in an arraycamera can be readily determined, therefore, the unknown quantity inestimating the scene dependent geometric shifts observed in the capturedimages is the distance between the array camera and different portionsof the scene. When a depth map and visibility information is generatedin accordance with the processes outlined above, the scene dependentgeometric shifts resulting from the depths of each of the pixels can bedetermined and occluded pixels can be ignored when performingsuper-resolution processing. In many embodiments, a confidence map isgenerated as part of the process of generating a depth map and theconfidence map is provided as an input to the super-resolution processto assist the super-resolution process in evaluating the reliability ofdepth estimates contained within the depth map when performing fusion ofthe pixels from the input images.

A process for generating a high resolution image using a light fieldcaptured by an array camera involving the generation of a depth map inaccordance with an embodiment of the invention is illustrated in FIG.17. The process 1700 involves capturing (1702) a light field using anarray camera and selecting (1704) a reference viewpoint that can beutilized to synthesize a high resolution image. In many embodiments, thereference viewpoint is predetermined based upon the configuration of thearray camera. In a number of embodiments, calibration information isutilized to increase (1706) the correspondence between captured images.In many embodiments, the correspondence between captured images involvesresampling the images. An initial depth map is determined (1708) andocclusions are determined and used to update (1710) the depth map. Inseveral embodiments, the process of detecting occlusions and updatingthe depth map is iterative.

In a number of embodiments, the depth map is utilized to generate (1712)information concerning the visibility of the pixels within the capturedlight field from the reference viewpoint. In several embodiments, aconfidence map is (optionally) generated (1713) with respect to thedepth estimates contained within the depth map and the depth map, thevisibility information, and/or the confidence map are provided (1714) toa super-resolution processing pipeline. In several embodiments, thesuper-resolution processing pipeline is similar to any of thesuper-resolution processing pipelines disclosed in U.S. patentapplication Ser. No. 12/967,807. The super-resolution processingpipeline utilizes information including the light field, the depth map,the visibility information, and the confidence map to synthesize (1718)a high resolution image from the reference viewpoint, which is output(1718) by the array camera. In several embodiments, the process ofsynthesizing a higher resolution image involves a pilot fusion of theimage data from the light field onto a higher resolution grid. Theresult of the pilot fusion can then be utilized as a starting point tosynthesize a higher resolution image using the super-resolution process.

As is described in U.S. patent application Ser. No. 12/967,807, theprocess illustrated in FIG. 17 can be performed to synthesize astereoscopic 3D image pair from the captured light field. Although aspecific process for synthesizing a high resolution image from acaptured light field is illustrated in FIG. 17, any of a variety ofprocesses for synthesizing high resolution images from captured lightfields involving the measurement of the depth of pixels within the lightfield can be utilized in accordance with embodiments of the invention.

While the above description contains many specific embodiments of theinvention, these should not be construed as limitations on the scope ofthe invention, but rather as an example of one embodiment thereof.Accordingly, the scope of the invention should be determined not by theembodiments illustrated, but by the appended claims and theirequivalents.

What is claimed is:
 1. A method of estimating distances to objectswithin a scene based upon a set of images captured from differentviewpoints using a processor configured by an image processingapplication, the method comprising: selecting a viewpoint of an imagefrom the set of images captured from different viewpoints as a referenceviewpoint; normalizing the set of images to increase the similarity ofcorresponding pixels within the set of images; determining depthestimates for pixel locations in an image from the reference viewpointusing at least a subset of the set of images, wherein generating a depthestimate for a given pixel location in the image from the referenceviewpoint comprises: identifying pixels in the at least a subset of theset of images that correspond to the given pixel location in the imagefrom the reference viewpoint based upon expected disparity at aplurality of depths; comparing the similarity of the correspondingpixels identified at each of the plurality of depths; and selecting thedepth from the plurality of depths at which the identified correspondingpixels have the highest degree of similarity as a depth estimate for thegiven pixel location in the image from the reference viewpoint;determining the visibility of the pixels in the set of images from thereference viewpoint by comparing the photometric similarity of pixelsfrom the set of images corresponding to a given pixel in the image fromthe set of images captured from the reference viewpoint, where pixelsfrom the set of images that correspond to a given pixel are determinedbased upon the depth estimate determined for the given pixel; and fusingpixels from the set of images based upon the depth estimates to create afused image having a resolution that is greater than the resolutions ofthe images in the set of images by: identifying the pixels from the setof images that are visible in an image from the reference viewpointusing the visibility information; applying scene dependent geometricshifts to the pixels from the set of images that are visible in an imagefrom the reference viewpoint to shift the pixels into the referenceviewpoint, where the scene dependent geometric shifts are determinedusing the depth estimates; and fusing the shifted pixels from the set ofimages to create a fused image from the reference viewpoint having aresolution that is greater than the resolutions of the images in the setof images.